What AI engineers are building

Advances in LLM inference, optimization, and automation continue to drive innovation in AI engineering

3 deep dives · 2 builder stories · 10 quick hits · 346 items scanned

LLM inferenceAI optimizationModel Context Protocol (MCP)AutomationGraphRAGRAG

Deep dives

GitHub

LLM inferenceEfficient deploymentEdge AI

FareedKhan-dev/kimi-k3-in-c

Fareed Khan has implemented a 2.78-trillion-parameter Kimi K3 model in C, achieving inference on a single CPU in 8.24 GB of RAM. This project demonstrates the potential for efficient LLM deployment on resource-constrained devices. The implementation is notable for its portability, requiring no BLAS, framework, or GPU.

Why it matters

This project showcases the feasibility of running large LLMs on limited hardware, which is crucial for edge AI applications. It also highlights the importance of optimizing LLMs for inference efficiency.

Try thisExplore the Kimi K3 implementation and experiment with optimizing other LLMs for resource-constrained devices.

View on GitHub↗

GitHub

LLM optimizationInference servingBenchmarking

patchy631/time-to-first-token

Patchy631 has created a 10-week roadmap for LLM inference serving and optimization. The project covers topics such as vLLM, SGLang, quantization, speculative decoding, and benchmarking. This resource provides a structured approach to improving LLM performance and efficiency.

Why it matters

Optimizing LLMs is essential for real-world applications, and this roadmap offers a comprehensive guide for achieving better performance and efficiency. By following this roadmap, developers can significantly improve their LLM-based projects.

Try thisFollow the roadmap and apply the optimization techniques to your own LLM projects.

View on GitHub↗

GitHub

LLM inferenceApple SiliconMoE

NeelM0906/Mference

NeelM0906 has developed a Swift and Metal-based MoE inference engine for Apple Silicon devices. The project achieves impressive performance, with Gemma 4 26B running in ~2 GB and Qwen 3.6 35B in ~1.45 GB. This engine also supports SSD-streamed experts and includes a native Mac app, CLI, and OpenAI-compatible server.

Why it matters

This project demonstrates the potential for efficient LLM inference on Apple Silicon devices, which is crucial for edge AI applications. The engine's performance and features make it an attractive solution for developers.

Try thisExplore the Mference engine and experiment with integrating it into your own projects.

View on GitHub↗

What people actually built

Hacker News

Show HN: Runthru – open-source Interactive Demos

Mark Tolson created Runthru, an open-source tool for creating interactive demos and walkthroughs for web-based software. The tool allows users to point it at a site and either let AI generate the demo or create it manually.

How it works

Runthru uses a combination of AI and user input to generate interactive demos. The tool is designed to be flexible and adaptable to different use cases.

Steal thisDevelopers can use Runthru to create engaging and interactive demos for their web-based software, improving user experience and adoption.

View on GitHub↗

Hacker News

Show HN: Texpile, an open-source desktop LaTeX editor with a visual mode

The Texpile team created an open-source desktop LaTeX editor with a visual mode, aiming to provide a more user-friendly experience for writing documents in LaTeX.

How it works

Texpile combines the power of LaTeX with a visual editor, allowing users to create and edit documents more efficiently. The tool also supports real-time collaboration and offline use.

Steal thisDevelopers can use Texpile to create and edit LaTeX documents more efficiently, and the tool's visual mode makes it more accessible to non-technical users.

View on GitHub↗

Quick hits

AirLLM 70B inference with single 4GB GPU ↗

Hacker News

AirLLM achieves impressive performance with a single 4GB GPU, demonstrating the potential for efficient LLM inference on resource-constrained devices.

moonshotai/Kimi-K3 ↗

Hugging Face

The Kimi-K3 model is trending on Hugging Face, with 9900 likes and 967,622 downloads, indicating its popularity and potential for real-world applications.

bybit-exchange/kaas ↗

GitHub

Kaas is an LLM knowledge-base compiler that turns scattered notes, docs, and transcripts into a queryable Markdown wiki, providing a valuable tool for knowledge management and retrieval.

lordx64/pentestkit ↗