Hunt for the optimal chunking strategy for document embedding

Source code

April 2025

When processing documents that exceed the context window of embedding models, practitioners employ different types of chunking strategies. Would changing the chunking strategy realistically affect the RAG performance? This evaluation compares three primary chunking methods (fixed-size, semantic, and sentence-based) across top embedding models (acc to MTEB leaderboard) to determine which approach yields optimal retrieval performance for different datasets. For more details, visit the repo.

Stream DS: A Distributed streaming platform

Source code

August 2024 - December 2024

This project was developed as part of a four-part Machine Problem (MP) series for the CS425 Distributed Systems course at UIUC.

For each of the MP, we built a different component of the distributed streaming system. The components include

• A distributed log querier (MP1)
• A failure detector built using the SWIM protocol (MP2)
• A distributed file system storage with replication and crash tolerance (MP3)
• A distributed stream processing engine (MP4)

The detailed architecture, design choices and performance reports for this project can be found in the source code repository. This project has been developed using go-lang and a lot of the concepts were inspired from MillWheel for the final MP. We achieved fault tolerance and re stabilization while also providing exactly once delivery semantics. All of the components were heavily tested and demo-ed on a cluster of 10 VMs.

Artificial Intelligence Machine Problems

August 2024 - December 2024

Completed a series of Machine Problems (MPs) for the CS440 Artificial Intelligence course at UIUC.

These MPs included everything from bayes nets, hidden markov models, reinforcement learning, to adversarial search algorithms and path planning for shape changing alien agents!

Beluga : An Interpreter for a self-made esoteric language

Source code

late Jan '23

I have always loved the idea of building my own programming language. So I started off trying to build a simple interpreter for a language in which you can directly code turing machines and execute them!

The language is written in pure C++ and follows the standard process of tokenization, parsing, abstract syntax tree generation, topological resource sorting and then execution. (does not use manually attached LLVM)

Please feel free to try it out on your machine or open an issue in the repository .The language is still nascent and I am working on adding more features to it.

One of the domains where this language can find universal usage outside education is in Regex. Many of the regex parsers used today are based on FSMs, and this language can be used to directly code a higher up, a TM. So in the near future I will be trying to create a regex api within the language.

A detailed description on how to install Beluga and run it can be found on the source code link

Rusty Vault : A Secure Password Manager written in Rust

Source code

August 2024

Do you store your passwords in excel sheets or word docs, unprotected? Do you use the same password for multiple accounts? Now is the time to stop all of that!

RustyVault is designed to be an exceptionally fast password manager, prioritizing swift user interactions and aiming to provide the same level of accessibility as accessing information from an Excel or Word document. Detailed implementation information can be found in the source code repository's documentation. RustyVault also ensures a high level of cryptographic security.

NexTech : Smart Document Capture Platform SIH 2022

Source code

July 2022 - August 2022

Secured 2 ^nd Place

In this project/hackathon me and my team developed a scalable document transfer pipeline solution for the Indian government

The problem statement involved everything from capturing the document without leaving any memory trace in the capturing device, running ML algorithms for enhancing the document quality, to finally designing a scalable server which can cater to millions of requests per day.

We developed a server-less architecture using AWS lambda and AWS API gateway to upload the images capable of handling around 10,000+ requests per second.Built a fully functional mobile application using flutter for capturing document images, editing them and uploading them.Also implemented a light weight image quality enhancer which runs on the server using canny edge detection with added customization

Research Project on Hybrid Neural Networks

Source code

2023

The future of Machine Learning Neural nets, specifically in the fields where reasoned accuracy is very important while taking sensitive decisions, is in the hybridization of neural networks and human engineered baseline features.

In this project we developed and tested various hybrid neural networks, and evaluated their performance on the basis of accuracy on classification of images containing uncontrolled fires (eg. wildfires) vs images containing controlled fires (eg. candles,fireplace)

The models which amalgamated human engineered features with its auto generated weights (unknown/blackbox) performed better and had a human baseline to classify an image incase it confronted any outlying data.

Discord Bot : Dazzle

Source code

2021 - 2022

I had started building this discord bot for my first year college class server, the main aim was to build some games which everyone could play against each other. (And kill time during online classes :p)

Very soon, the bot could run many different games and had a lot of api integrations for different services (eg. OpenAI text completion api (2021)). It also had a fully functional economy system with a bank and a shop. It was used actively on 10+ servers.

It is written using discord.py library, and is no longer hosted on Heroku.