Here’s a sample of the JetsonHacks Newlsetter. If you find the content interesting, please subscribe! Keep up to date, and get some industry insight at the same time. Click here to subscribe. BTW, here’s me at GTC many moons ago:
Hi there!
I had to involuntarily take a little time to be distant in August. Let’s make the JetsonHacks Newsletter more interesting as we go towards the end of the year!
Jetson News
The big news in Jetson land is the announcement of the in-person NVIDIA GTC Conference March 18-21 of 2024 in San Jose, California. https://www.nvidia.com/gtc/ The last one was in 2019, and unless you have been hiding under a rock (like I enjoy doing) very much has changed in the whole NVIDIA ecosystem world. I may have been tricked into attending.
NVIDIA has doubled in size from ~ 13K to 26K employees in that time. At this point, I don’t even know what to expect. It might be magical, a little scary, or even fun. I dread the last one a little. I’m planning to be stoic and brave my way through it. More details as I learn them. I certainly look forward to seeing you there!
Jetson Generative AI Playground
The Jetson group has opened up a new playground! If you’ve been watching the way excellent dusty-nv account on Github (https://github.com/dusty-nv) you know that the jetson-containers and jetson-inference repositories are pure gold. The Jetson Generative AI Playground is a new idea, a website to provide tutorials on how to use generative AI on the Jetson. At the beginning, there are three great tutorials certainly worth studying. More on the way!
The Turing Pi 2 Clusterboard looks like a reasonably priced mini-ITX cluster board that can support up to 4 Jetson modules of NX/Nano format. I haven’t worked with one of these yet, but the idea of 4 Jetsons federated together is certainly interesting. There are all sorts of interesting ideas to explore on this kind of setup.
If you’re into hardcore drones, Flyby is introducing the Flyby F-11. They’re an interesting group of people. Money quote: ‘We’re a group of builders from Yale, Princeton, NASA JPL, and Anduril working together to redefine what American drones can accomplish.’ It’s a little pricey ($16.5K) but it has all the good pro bits. Jetson Orin inside.
Antmicro
Antmicro has been working with Jetsons from the beginning. In August they announced that they were able to create and release an open source board implementing a PCIe-Thunderbolt bridge. The on-board PCIe x4 exposed by Jetson NX/Nano form factor SoMs is used to facilitate the bridge. An alternative solution could be to use a 10Gb Ethernet controller. Antmicro has a solution for this also. There are a bunch of interesting applications that can benefit by having more and faster pipes.
Antmicro does some really interesting things with Jetsons. There are a lot of fun, serious projects on the site. Read the article, browse the website and get some ideas. Certainly worth a look!
Generative AI
Here’s the usual insight part of the Newsletter. I’m sure you’ve thought about this topic quite a bit already.
As you know, many companies are rolling in trucks full of money and funneling it into machine learning infrastructure. The majority of those trucks are going straight to NVIDIA in the hopes that they may be able to get the latest and greatest. No fools, NVIDIA is happy to help other people get rid of the dirty, filthy lucre.
In exchange the lucky few (as the current machine learning monster cards are in short supply) receive world class performing machines designed for training machine learning models. For example, Tesla dropped ~ $300M so they could get to even more training. We all feel we know what Tesla may be working on.
It’s always hard to guess what those types of big numbers mean. Such deals usually inflate the project to include everything but the kitchen sink. However, NVIDIA reported $10B of data center revenue last quarter. That’s not the type of coin that you find in the sofa. The point is that the money funnel is something worth thinking about.
Here’s the Thing
There’s a scramble to acquire this high-end hardware, and employ the few people who can really take advantage of it. It’s the wild west. It’s been a long time since the tech industry has had this type of gold rush feel.
You and I know something most others don’t. All that money that’s funneling into machine learning training? It produces a product which must utilize inferencing at some point. The second gold rush will be to get the models produced from the big money investment running at a decent price point at scale.
This might mean custom hardware. For example, Tesla will use their models in their vehicles and humanoid robots. Both of these use Tesla custom processors. We know startups and established players alike have or are designing their own inferencing hardware. Google with their TPU, for example.
We also know the some of the inferencing will come from cloud based services. The data racks will be humming!
Those are some of the nets catching the money as it comes out the bottom of the funnel. Take those money catchers out of the mix. There is a great pull towards data privacy. Many companies want their data to be separate from others, and auditable. The idea of having data and models local is more than appealing to many. A big question is where does this happen?
Does it mean in ‘private’ data centers, on-premises computing, or edge computing? Likely a combination of all of these. What about personal users, people who aren’t doing paid work for someone else?
Many will benefit from task specific machine learning models built into products. For example, this is already happening on phones with built in machine learning hardware like the Apple iPhone. The ML helps in computational photography and different aspects of the graphics pipeline among other tasks.
Many people will use free or relatively inexpensive machine learning services. Today ChatGPT and Midjourney come first to mind. The more tech-savvy might build their own. Make use of a personal computer or laptop with capable programmable graphics/ML blocks. Think NVIDIA graphics card or Apple Silicon based systems.
What about the edge, companies and personal use alike? What small machines have enough horsepower to run these models today? You beat me to it.
The Jetson Orins have enough horsepower to run the models, and it’s unlikely they will have much serious competition for some time. You know how long it takes to create new chips and design systems that take advantage of them.
Another point is that the models and their formats are malleable right now. It’s unlikely that models as we know them today will be reified into a state which special purpose hardware can execute easily. The Jetsons have an advantage in that they are more general purpose/configurable, and use the same architecture as the machines that were used to train the models. Another factor is that there is a shortage of silicon wafers and world wide chip production capacity currently.
Crystal Ball
Figure that the Jetsons will get better over time. Since the Orin introduction, we’ve seen an increase of 84% in the machine learning benchmarks from software alone. Hardware iterations after that will increase performance further.
The top of the line Jetson AGX Orin Development Kit is not inexpensive. It’s $2K. The other side of that coin is that you can run applications like a Llama LLM along with speech recognition and text to speech on it. Or generate art using Stable Diffusion. Or any other of the hundreds of machine learning models now available. Some of this can be done on an Orin Nano, much more on an Orin NX 16GB. A 64GB AGX Orin can handle several tasks at once. Memory is a big deal in this arena.
One way to look at it is that the Jetsons are too expensive. Another, more insightful, way is to realize that here are applications that could not have been done at any cost 5 years ago by mere mortals. Now you can do them for the cost of a high end graphics card. Take advantage.
You have a multi-year head start on everyone at this point. Take time to understand the capabilities, gain experience, and figure out what new magic to invent. Then go do it!
The post JetsonHacks Newsletter Sample – August/September 2023 appeared first on JetsonHacks.