At the same time LLMs are changing the world, we have witnessed fast progress in AI image and video generation. AI models are now able to synthesis high quality, high fidelity and high resolution images via text prompting. Mainstream generative model architectures have shifted from VAEs, flows and GANs to diffusions and transformers. In this post I take notes of several vision models. It may be regularly updated to reflect latest research development....
In this post I will walk through the transformer layer and several improvements over this architecture that are commonly employed in many popular open source large language models (LLMs) today, for example Llama. Discussed include SwiGLU and RMSNorm layers, RoPE and ALiBi position embeddings; and finally Flash Attention for scaling attention calculation to long sequences. We will use Llama source code as example implementation, and toward the end I’ll go through the rest of Llama’s source code....
On April 11, 2022, I wrote the following: “While current models may still have limited capacity, I’m optimistic that, we should see better and better AI models coming out in the future, and one day machines can have enough intelligence to handle tasks and jobs we do today.” 7 months later, OpenAI released ChatGPT, the performance of which has stunned the public. It can chat, write stories, debug code…there hasn’t been a chatbot that is so intelligent and so fluent in dialogues with human....
Previously, I use Jekyll to build my website. After careful consideration, now I decide to switch to Hugo. The main reason is straightforward: Speed. Jekyll does not scale It all looks nice and great when you look at Jekyll’s official website. My initial reasons for choosing Jekyll include its integration with GitHub Pages, good documentation, wide range of themes and plugins. But when I have some 20 posts, it takes me 5~6+ seconds to build the site, and it takes around 4 seconds for reflecting edits in live preview, which is almost unbearable!...
See here 👉https://lifeitech.github.io