Any LLM, RAG and production deployment PT 2

Context

Since my very first blog about LLM, RAG and production it was more a quick showoff about how prod I was my self to run locally a small LLM (LLama3.1) in my gaming PC, specs in PT1.

this time I want to add more context

From day 1 I always knew what I want to accomplish with these new technologies and what I did not want to.

I wanted to understand and build something using core elements no APIs or external services.

So NO OpenAI API, LangChain, chroma, NO external services.

FULL python, that's what you notice first time reading about llms and Pytorch in this case because Llama3.1

You can see the date Jun 17 2025, still no claude code only OpenAI web chat for questions and answers.

I have to confess I got the interest on AI stuff very late from a good friend because to me it was more for scientists. I did not feel comfortable talking about AI since I'm not a scientist but there is room for all if you look closely.

first steps python virtual environments and pip package manager, I need to run code on my 4070 TI super and install NVIDIA drivers so many problems happened but I made it and spend long time playing with it running python code directly to the Vcard chip.

My focus was running python code libraries and learning from Llama3.1 code and pytorch examples.

• Why is it for?
• How can this be useful?
• How to send request to a web server like OpenAI?
• Who is Guillermo Quezada from Colima?... I asked to LLM no data.

That's how I started into RAG research my self.

Around April 2026 news about Qwen3.6 doing magic locally.

HuggingFace appeared one day to stay, it's like Github for AI opensource work, datasets, embeddings, llms.

I wanted to run it, first big wall OOM errors vRam is very important and Vllm/Slang for inference.

By reading HF repositories you can get pre-requisites to run stuff and immediately notice ollama was not for me it's a tool for low resources machines not mine I will go with VLLM.

Purchased 2nd vCard 5070Ti April 2026

I have now 32GB vRAM for 1k each total 2kUSD instead to be fucked by 5090 4.5KUSD same vRAM. then summarizing many days I started to generate production ready code with Qwen3.627B local and OpenCode using local inference and my power billing noticed.

RAG research continue.. now there is also AGENTIC RAG and AWS BEDROCK

June 2026 Company provides claude code, after many weeks delaying it is my first time using it, I did not want to stop using Qwen3.627B and opencode but it turns out claude code is excellent and free in my case.

As you can see I started very late using llms for real work or building software while everybody was already sharing Claude Code tutorials and tips I preferred to focus on LLMs background topics so by the time I started using AI coding agents they already were very good and easy to prompt.

RAG

You want Retrieval-Augmented Generation to integrate new knowledge sources to LLMs like your company or personal documents, this is just another use for LLM features image a Customer Desktop Agent that can reply to questions about some product or query DB to get detailed reports or AI Agents consuming private/public data.

What is vLLM

vLLM is an open-source high-throughput, low-latency inference and serving engine for large language models. It implements PagedAttention for efficient KV-cache memory management, continuous batching, chunked prefill, tensor and pipeline parallelism, quantization (AWQ, INT4, FP8 KV cache), and an OpenAI-compatible HTTP API. It can be run directly from Python or as a server reachable with any OpenAI client.

Starting the OpenAI-compatible server

Activate the environment first:

  source .venv/bin/activate
    

Multi-GPU launch with tensor parallelism

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,2
export FLASHINFER_CUDA_ARCH_LIST="10.0 8.9"

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
python -m vllm.entrypoints.openai.api_server \
  --model Qwen3.6-27B-AWQ \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --language-model-only \
  --dtype float16 \
  --max-model-len 130144 \
  --gpu-memory-utilization 0.90 \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --enable-chunked-prefill \
  --max-num-seqs 32
    

Querying the server

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "Qwen3.5-9B-AWQ",
  "messages": [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Explain dark energy to me and elaborate on it."}
  ],
  "temperature": 0.7,
  "max_tokens": 4096
}' | jq ".choices[0].message.content" | pandoc -t markdown_strict
    

Local LLM Memory: Documents, Embeddings, and pgvector

A local LLM is only as useful as the context you give it. Model weights supply the reasoning, but your own documents supply the facts. The usual way to bridge the two is retrieval-augmented generation: convert documents into embeddings, store them in a vector database, retrieve the closest passages for a question, and place those passages in the model’s prompt.

This post walks through a small, fully local version of that pipeline. It uses the sentence-transformers/multi-qa-MiniLM-L6-cos-v1 model from Hugging Face, PyTorch for encoding, and PostgreSQL with the pgvector extension for storage and search. Nothing leaves the machine: no hosted embedding API, no external vector service.

Why local vectors?

Keeping embeddings local has three practical advantages:

• Privacy. The document text and its vector representation stay in your own PostgreSQL instance.
• Cost. Encoding runs on your CPU or GPU, and search runs inside a database you already operate.
• Control. You can inspect the schema, run SQL joins, back up the data, and change the model without waiting on a vendor.

The trade-off is that you own the operational details: model dimensions, schema design, chunking, indexes, and query semantics.

The embedding model

Embedding model turns text into a fixed-length float vector so that semantically similar text lands close together. In RAG you embed chunks once into an index, embed the query at search time, and rank by cosine similarity (vectors normalized). Quality depends on the model, chunk size, context window, language coverage, and whether it expects query/passage prefixes. multi-qa-MiniLM-L6-cos-v1 is a 2021-era model, English, trained for QA retrieval — fine as a baseline, but limited by today's standards (short context, weak multilingual, low MTEB).

multi-qa-MiniLM-L6-cos-v1 is a compact sentence-transformer trained for question-answering retrieval. It maps text to a dimensional vector. That number matters more than it might seem, because it has to match the database column exactly.

The encoding logic in transformer.py is deliberately small:

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output.last_hidden_state
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(
        token_embeddings.size()
    ).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(
        input_mask_expanded.sum(1), min=1e-9
    )

def encode(texts):
    encoded_input = tokenizer(
        texts, padding=True, truncation=True, return_tensors="pt"
    )
    with torch.no_grad():
        model_output = model(**encoded_input, return_dict=True)

    embeddings = mean_pooling(
        model_output, encoded_input["attention_mask"]
    )
    embeddings = F.normalize(embeddings, p=2, dim=1)
    return embeddings

The tokenizer pads short inputs and truncates long ones. Mean pooling averages only the real tokens, using the attention mask so padding does not dilute the result. The final L2 normalization puts every vector on the unit sphere, which makes dot-product and cosine similarity equivalent and keeps distances well-behaved.

get_embedding() then flattens the tensor and converts it to a plain Python list so psycopg2 can send it to PostgreSQL:

def get_embedding(text):
    return encode(text).numpy().flatten().tolist()

The schema, and a dimension mismatch worth remembering

The first version of the table declared vector(1536):

CREATE TABLE IF NOT EXISTS documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding vector(1536)
);

I verified the live database with psql:

psql -d vectors -c '\d+ documents'
psql -d vectors -c '\dx'
psql -d vectors -c 'SELECT count(*) FROM documents;'

The relevant details:

Item	Value
Table	public.documents
id	integer NOT NULL DEFAULT nextval('documents_id_seq')
content	text
embedding	vector(1536)
Primary key	documents_pkey btree on id
Rows	8, all with 1536 dimensions
pgvector version	0.8.7

Inserting embeddings

embedding = get_embedding(content)
cur.execute(
    "INSERT INTO documents (content, embedding) VALUES (%s, %s::vector)",
    (content, embedding),
)
conn.commit()

Searching: pgvector versus PyTorch

The retrieval query in transformer.py is:

def search_documents(query, limit=4):
    query_embedding = get_embedding(query)
    cur.execute(
        """
        SELECT content, embedding <-> %s::vector AS distance
        FROM documents
        ORDER BY distance
        LIMIT %s
        """,
        (query_embedding, limit),
    )
    return cur.fetchall()

In pgvector, <-> is Euclidean (L2) distance. Because both the stored vectors and the query vector are L2-normalized, ordering by L2 distance produces the same ranking as ordering by cosine similarity. If you want to make the intent explicit, use the cosine distance operator <=>, or <#> for negative inner product.

The alternative is to do everything in memory with PyTorch:

scores = torch.mm(query_emb, doc_emb.transpose(0, 1))[0].cpu().tolist()

Both approaches measure similarity, but they serve different purposes:

	pgvector <->	torch.mm
Data location	Persistent, in PostgreSQL	In-memory tensors
Scale	Large collections, disk-backed	What fits in RAM or VRAM
Acceleration	ANN indexes (HNSW, IVFFlat)	CPU/GPU parallelism
Metric	L2 (also cosine and inner product)	Dot product (cosine if normalized)
Integration	SQL, joins, transactions	Python/ML pipelines
Typical use	Production retrieval	Experiments, evaluation, reranking

In practice, the two complement each other. pgvector narrows millions of documents down to a handful of candidates; PyTorch handles custom scoring or reranking on that short list. For a database this small, either approach is instantaneous, but the SQL version is the one that scales without loading the corpus into memory.

Documents are not chunks

A second design question is what actually goes into the content column. Embedding an entire long document produces one vague vector and wastes the model’s limited context window. Retrieval quality improves when documents are split into smaller passages, typically 256 to 512 tokens, with a 50 to 100 token overlap so context is not lost at boundaries.

A simple sentence-based chunker is enough to start:

def chunk_document(document, chunk_size=5):
    sentences = sent_tokenize(document)
    chunks, current_chunk, current_size = [], [], 0

    for sentence in sentences:
        sentence_tokens = len(sentence.split())
        if current_size + sentence_tokens > chunk_size:
            chunks.append(" ".join(current_chunk))
            current_chunk = [sentence]
            current_size = sentence_tokens
        else:
            current_chunk.append(sentence)
            current_size += sentence_tokens

    if current_chunk:
        chunks.append(" ".join(current_chunk))
    return chunks

Each chunk should also carry metadata: a document ID, a chunk position, perhaps a title, source, and timestamp. The current documents table stores only content and embedding; a production version would add doc_id, chunk_id, and a metadata jsonb column, and insert one row per chunk.

Demo

Insert data

Remember - Who is Guillermo Quezada from Colima? so I decided to use my CV as new source to insert chunks in vector DB and another fake Bob cv for testing distances.

Query data

Distance metrics define what 'similar' means when comparing embeddings. They are only used to rank stored vectors, not to generate answers.

Response Generation

Opencode vllm local server and the smallest model Qwen3.59B since I don't have mounted second vCard right now LOL

This is just the begging there is alot more to do for production and real use cases:

• Build an evaluation set first.
• Improve chunking next
• Add hybrid retrieval and reranking.
• Improve query handling.
• Improve generation prompts.
• Fine-tune only after measuring.

Notes

RAG is not just a query-and-answer pattern but a grounding and data-access layer that feeds models and agents the right context to plan, route, act, and orchestrate—making it foundational infrastructure for far more than chat.

References

• Github Guillermo Quezada - torukMnk
• HuggingFace Guillermo Quezada - torukMnk
• Zero to One: Learning Agentic Patterns
• pgvector Tutorial
• Llama 3.1 RAG Tutorial
• OpenAI Embeddings Guide

Test