This isn’t inherently better or worse. It’s a different set of tradeoffs: swap SaaS convenience for control, predictability, and (usually) lower costs as you scale.

What Forgejo Provides

Forgejo started as a soft fork of Gitea in 2022 over governance concerns. In February 2024, it became a hard fork under Codeberg e.V., achieving full independence from its upstream. The latest stable release is v13.0 (October 2025), bringing WebAuthn support and 15% memory optimizations.

Core functionality:

• Git repository hosting

• Pull request and code review workflow

• Issue tracking with labels and milestones

• Built-in CI/CD (Forgejo Actions)

• Package registry (Docker, npm, Maven, PyPI)

• Git LFS support

• OAuth/LDAP/SAML authentication

• Federation for multi-instance collaboration

• AI code suggestions (opt-in plugins)

The interface resembles GitHub. Teams familiar with GitHub’s workflow navigate Forgejo without training.

Cost Structure

GitHub Teams: $5/user/month ($60/year per user) GitLab Premium: $32/user/month ($384/year per user)

For 10 users:

• GitHub: $600/year

• GitLab: $3,840/year

Forgejo on a VPS:

• Hetzner CX21: €4.85/month (~$52/year)

• Digital Ocean Droplet (2GB): $12/month ($144/year)

These VPS prices work for 10 users or 100 users. No per-seat scaling.

Hidden costs:

• Setup time: 4-6 hours

• Ongoing maintenance: 2-4 hours/month

• Backup storage: $1-5/month

Technical Requirements

Minimum specs:

• 1GB RAM (tight, but usable)

• 2 CPU cores

• 20GB storage + repository sizes

• Linux server (Ubuntu, Debian)

Recommended for production:

• 2GB RAM

• 2-4 CPU cores

• Storage: 50GB base + 1.3x repository sizes

• SSD strongly recommended

Database options:

• SQLite: Simple, single file. Works until concurrent operations slow down (10-20 active users).

• PostgreSQL: Better concurrency. Migrate when SQLite bottlenecks.

Installation

Docker Compose setup with health checks:

services:
  forgejo:
    image: codeberg.org/forgejo/forgejo:9-rootless
    container_name: forgejo
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - FORGEJO__database__DB_TYPE=postgres
      - FORGEJO__database__HOST=db:5432
      - FORGEJO__database__NAME=forgejo
      - FORGEJO__database__USER=forgejo
      - FORGEJO__database__PASSWD=your_password_here
      - FORGEJO__server__ROOT_URL=https://git.yourdomain.com
      - FORGEJO__server__SSH_PORT=222
    volumes:
      - ./data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  db:
    image: postgres:17-alpine
    environment:
      - POSTGRES_USER=forgejo
      - POSTGRES_PASSWORD=your_password_here
      - POSTGRES_DB=forgejo
    volumes:
      - ./postgres:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U forgejo"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

Notes:

• Rootless image for simpler permissions

• SSH on port 222 avoids host conflicts

• Health checks ensure service readiness

• PostgreSQL 17 for latest performance improvements

Start: docker-compose up -d

First run launches web installer at

http://your-server:3000

. Configure admin credentials, server URL, and email settings.

Reverse Proxy and SSL

Forgejo serves HTTP on port 3000. Production needs HTTPS.

Caddy (automatic SSL):

git.yourdomain.com {
    reverse_proxy forgejo:3000
}

Nginx:

server {
    listen 443 ssl http2;
    server_name git.yourdomain.com;
    
    ssl_certificate /etc/letsencrypt/live/git.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/git.yourdomain.com/privkey.pem;
    
    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Get certificates: certbot --nginx -d git.yourdomain.com

Basic Usage

Repositories:

Organizations group repositories and manage team permissions. Create organizations for team projects, user accounts for personal repos.

Clone with custom SSH port:

git clone ssh://git@git.yourdomain.com:222/org/repo.git

Pull Requests:

Standard GitHub workflow:

1. Branch or fork

2. Push changes

3. Open pull request

4. Inline code review

5. Merge when approved

Issues:

Support Markdown, labels, milestones, assignees, attachments. Reference commits: fixes #123 creates automatic links.

Webhooks:

POST JSON to external endpoints on push, PR, issue, or release events. Configure in repository settings for CI/CD triggers.

Federation:

Connect multiple Forgejo instances for cross-instance collaboration. Users on one instance can interact with repositories on another. Enable in admin settings.

CI/CD: Forgejo Actions

GitHub Actions-compatible. Run your own runners.

Runner Setup:

services:
  runner:
    image: code.forgejo.org/forgejo/runner:3.3.0
    environment:
      - FORGEJO_INSTANCE_URL=https://git.yourdomain.com
      - FORGEJO_RUNNER_REGISTRATION_TOKEN=your_token
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped

Token from: Site Administration → Actions → Runners

Workflow Example:

.forgejo/workflows/test.yml:

name: Test
on: [push, pull_request]

jobs:
  test:
    runs-on: docker
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: pip install -r requirements.txt
      - run: pytest

Most GitHub Actions work without modification.

AI Code Suggestions

V13 introduces opt-in AI plugins for code completion and suggestions. Configure via admin panel. Integrates with OpenAI API or self-hosted models. Privacy-focused: code never leaves your infrastructure if using local models.

Enable in: Site Administration → AI Settings

Backup Strategy

Critical data:

1. Repositories (./data volume)

2. PostgreSQL database

3. Configuration (app.ini)

Daily Backup:

#!/bin/bash
BACKUP_DIR="/backups/forgejo"
DATE=$(date +%Y%m%d_%H%M%S)

# Database backup (no downtime)
docker-compose exec -T db pg_dump -U forgejo forgejo | gzip > $BACKUP_DIR/db_$DATE.sql.gz

# Repository backup
tar -czf $BACKUP_DIR/data_$DATE.tar.gz ./data

# Retention: 7 days
find $BACKUP_DIR -mtime +7 -delete

Off-site:

rclone sync /backups/forgejo s3:bucket-name/forgejo

Test restores regularly.

Updates

Updates every 4-8 weeks.

docker-compose pull
docker-compose up -d
docker-compose logs -f forgejo

Downtime: 30-60 seconds.

Security:

• Enable 2FA for admins

• SSH key-based auth only

• Strong database passwords

• Keep host OS updated

When to Self-Host

Good fit:

• You manage servers already

• Stable team size

• Data sovereignty requirements

• Infrastructure control preference

Poor fit:

• No operational expertise

• Rapid unpredictable growth

• Critical uptime without HA capability

• Small teams focused on product

Getting Started

Evaluation:

1. Deploy local instance

2. Create test repository

3. Try workflows (clone, commit, PR)

4. Test Actions

5. Practice backup/restore

Resources:

• Docs: forgejo.org/docs

• Forum: codeberg.org/forgejo/discussions

• Matrix: #forgejo:matrix.org

• Video: Forgejo Tutorial by Awesome Open Source

Bottom Line

Forgejo trades SaaS convenience for infrastructure control. You eliminate per-seat costs and external dependencies at the cost of operational responsibility.

The software is stable. Choose based on whether managing it aligns with your team’s capabilities, not ideology.

Why Compare vLLM with HuggingFace Transformers?

HuggingFace Transformers is the de facto standard library for working with large language models. It provides an accessible, well-documented interface that most ML practitioners use for training, fine-tuning, and inference. The library has facilitated access to state-of-the-art models and serves as the foundation for countless AI applications.

However, HuggingFace was designed primarily as a research and development tool, not as a production inference engine optimized for serving real users at scale. When deploying LLMs in production—especially for applications serving multiple concurrent users—HuggingFace’s straightforward approach becomes a significant bottleneck.

• Each request is processed sequentially

• GPU memory is allocated statically regardless of actual usage

• There’s no optimization for common patterns like shared system prompts.

These limitations compound quickly: a system that feels responsive during development becomes frustratingly slow under real-world load. This is precisely where vLLM enters the picture. vLLM is purpose-built for high-throughput LLM serving in production environments. It introduces several architectural innovations that fundamentally change how inference is performed:

• PagedAttention for efficient memory management,

• Continuous batching for handling concurrent requests dynamically, and

• Prefix caching for scenarios with repeated prompt patterns.

The benchmark validates whether these claimed improvements translate to real-world performance gains using a fine-tuned Qwen 2.5 7B model. The following 5 tests were conducted :

[1] Throughput Comparison (Sequential vs Batched vs vLLM)

This is the fundamental comparison that answers the core question: how much faster is vLLM? We measured how many tokens per second each approach can generate when processing 20 identical prompts. Throughput directly translates to cost and user experience. Higher throughput means serving more users with the same hardware, reducing infrastructure costs, and delivering faster responses. In a production environment, throughput determines how many concurrent users your system can handle before response times become unacceptable.

The results are striking and validate vLLM’s core value proposition. HuggingFace sequential processing—the default approach most developers use when following tutorials and documentation—achieved only 40.9 tokens per second. vLLM delivered 868.5 tokens per second, a 21.2x improvement. Even compared to HuggingFace’s batched inference (which requires additional code complexity and careful tuning), vLLM was 1.46x faster while being simpler to deploy and configure.

[2] Maximum Concurrent Sequences (KV Cache Efficiency)

This test pushed the system to handle increasing numbers of simultaneous requests (1, 10, 25, 50, 100, 150, 200) to identify the breaking point and understand scaling behaviour. Real-world applications don’t serve one user at a time. A local business assistant might hundreds of users querying simultaneously during peak hours. The ability to handle concurrent requests without crashing, running out of memory, or degrading severely determines whether the system is genuinely production-ready or merely a demo.

vLLM successfully handled all 200 concurrent sequences without failure. More importantly, throughput increased with concurrency—a counterintuitive result that demonstrates vLLM’s efficient batching strategy. The system achieved peak throughput of 3,184 tokens per second at maximum concurrency, nearly 70x faster than single-request HuggingFace performance. This remarkable scaling is possible because of PagedAttention, vLLM’s innovative memory management system. Traditional transformers allocate fixed memory blocks for each sequence’s key-value cache, leading to fragmentation and waste when sequences vary in length. PagedAttention allocates memory in small pages on-demand, similar to how operating systems manage virtual memory. This eliminates waste and allows significantly more sequences to run concurrently within the same GPU memory budget.

[3] Prefix Caching Impact

This test compared performance with and without prefix caching enabled, using 100 queries that shared a common system prompt—a scenario typical for chatbot and assistant applications. Production LLM applications typically use system prompts—detailed instructions that define the assistant’s behavior, personality, and constraints. For the fine-tuned model, every single query includes context about being a “helpful AI assistant for local business discovery ” that understands Hindi, English, Marathi, and Hinglish. Processing this identical prefix repeatedly for every user query wastes substantial computation.

Enabling prefix caching delivered a 32% speedup with zero code changes required. For the Discovery model’s specific use case—where every query shares the same system prompt defining its locality-focused, multilingual behavior—this optimization directly reduces latency and operational cost. Over thousands of daily queries, this improvement compounds into significant infrastructure savings and noticeably snappier user experiences.

[4] Concurrency Scaling

This test systematically measured how throughput and latency change as concurrent requests increase from 1 to 100. Understanding the scaling curve helps with capacity planning and setting realistic expectations. Does performance degrade gracefully under load, or does it collapse catastrophically? At what point do you need additional hardware?

The results demonstrate excellent scaling characteristics that defy naive expectations. Throughput increased nearly linearly with concurrency up to about 50 users, then continued improving at a diminishing but still positive rate. Remarkably, per-request latency dropped from 2.8 seconds (single request) to an effective 42ms per request when processing 100 concurrent requests in 4.2 seconds total.

This counterintuitive behavior demonstrates continuous batching working exactly as designed: the GPU stays fully utilized processing multiple requests simultaneously, and individual requests benefit from efficient batched matrix operations.

[5] Chunked Prefill

This test evaluated chunked prefill performance with long-context prompts containing approximately 1,128 tokens each. Long prompts can cause latency spikes as the system must process the entire input before generating any output tokens. Chunked prefill addresses this by breaking prefill computation into smaller pieces, reducing time-to-first-token and improving perceived responsiveness.

For this moderate context length, chunked prefill showed minimal difference (~1%). This optimization becomes substantially more important with very long contexts exceeding 4,000 tokens. For typical Discovery model queries, which involve relatively short user questions, this feature is less critical but remains available when processing longer documents or conversation histories.

Conclusion

The benchmark comprehensively validates vLLM’s performance claim against HuggingFace Transformers.

For the fine-tuned Qwen2.5-7B model specifically, these results demonstrate that a single A5000 GPU can serve a substantial user base effectively. The prefix caching optimization aligns perfectly with the model’s architectural use case, where every query shares the same system prompt defining its local business assistant persona.

The choice to benchmark against HuggingFace Transformers provides a meaningful baseline that practitioners universally understand. vLLM isn’t incrementally better—it’s an order of magnitude faster, representing the difference between a functional prototype and a genuinely production-ready system capable of serving real users reliably.

What is Exploratory Data Analysis?

Exploratory Data Analysis is an approach for examining datasets and to summarize their main characteristics, often using visual methods. Pioneered by statistician John Tukey in the 1970s, EDA emphasizes the importance of looking at data before making assumptions or building models. Unlike confirmatory analysis, which tests specific hypotheses, EDA is an open-ended process of discovery.

The primary objectives of EDA include:

• Understanding the structure and distribution of data

• Detecting outliers and anomalies

• Identifying patterns, trends, and relationships between variables

• Testing underlying assumptions

• Selecting appropriate models for further analysis

EDA is not a rigid set of procedures but rather a state of mind that encourages curiosity and thorough investigation. It asks questions like: What does the data tell us? What doesn’t it tell us? What patterns emerge? What surprises exist?

The Power of Data Visualization in EDA

Data visualization transforms abstract numbers into concrete visual representations, making complex data accessible and understandable. Our brains process visual information a lot faster than the text, making visualization an indispensable tool for EDA. Well-designed visualizations can reveal trends, outliers, and relationships that might take hours to discover through statistical analysis alone.

Key visualization techniques in EDA include:

Histograms and Distribution Plots: Reveal the shape and spread of data

Scatter Plots: Uncover relationships between variables

Box Plots: Identify outliers and compare distributions across groups

Correlation Heatmaps: Display relationships between multiple variables

Time Series Plots: Track changes over time

Animated Visualizations: Show evolution of patterns across dimensions

Understanding the Gapminder Dataset

The Gapminder dataset, curated by the Gapminder Foundation, provides a fascinating lens through which to explore global development trends. It contains data for 142 countries from 1952 to 2007, tracking three critical metrics:

Life Expectancy

This metric measures the average number of years a person is expected to live at birth. In our dataset, life expectancy ranges from a sobering 23.6 years (Rwanda, 1992, during the genocide) to an impressive 82.6 years (Japan, 2007). This dramatic range immediately tells a story about global health disparities and the impact of conflict on human welfare.

Population

Population figures range from 60,011 (Sao Tome and Principe) to over 1.3 billion (China, 2007). This metric helps us understand demographic pressures, economic potential, and resource allocation challenges facing different nations.

GDP per Capita

Measured in international dollars, GDP per capita ranges from $241.17 to $113,523.13 (Kuwait during an oil boom). This economic indicator reveals the vast wealth inequality between nations and provides context for understanding quality of life differences.

Key Statistical Metrics in EDA

Measures of Central Tendency

• Mean : The average provides a quick snapshot but can be skewed by outliers

• Median : The middle value is more robust to extreme values

• Mode : The most frequent value, particularly useful for categorical data

For the Gapminder dataset, comparing mean and median GDP per capita reveals positive skewness—a few wealthy nations pull the average higher than the median, indicating wealth concentration.

Measures of Dispersion

• Standard Deviation : Quantifies variation around the mean

• Range : The difference between maximum and minimum values

• Interquartile Range (IQR) : The spread of the middle 50% of data, robust to outliers

• Coefficient of Variation (CV) : Enables comparison of variability across different scales

The CV is particularly valuable when comparing metrics with different units. For instance, while population has enormous absolute variation, its CV reveals whether this variation is proportionally larger than life expectancy’s variation.

Percentiles and Quartiles

Percentiles divide data into 100 equal parts, providing detailed distribution information. The 25th, 50th (median), and 75th percentiles are especially important, forming the basis of box plots and revealing data symmetry or skewness.

Correlation Analysis

Correlation coefficients (ranging from -1 to +1) measure the strength and direction of linear relationships between variables. In the Gapminder data, life expectancy and GDP per capita show strong positive correlation (typically > 0.7), suggesting that wealthier nations tend to have longer-lived populations. However, it’s crucial to remember that correlation does not imply causation.

Advantages of Systematic EDA

Early Problem Detection

EDA reveals data quality issues before they corrupt analysis results. Missing values, incorrect data types, duplicate records, and inconsistent formatting become apparent through summary statistics and visualizations.

Insight Generation

Unexpected patterns often emerge during EDA. For example, examining the Gapminder data might reveal that some countries experienced dramatic life expectancy drops during specific years, prompting investigation into historical events like wars or epidemics.

Communication Enhancement

Visualizations transcend language barriers and technical expertise levels. A well-designed animated scatter plot showing the evolution of life expectancy versus GDP over time can convey decades of development history in seconds, making data accessible to stakeholders at all levels.

Model Selection Guidance

Understanding data distributions, relationships, and outliers helps select appropriate analytical methods. For instance, discovering non-linear relationships between variables suggests the need for polynomial regression rather than simple linear models.

Common Pitfalls in Data Analysis

Confirmation Bias

Analysts sometimes unconsciously seek patterns that confirm pre-existing beliefs while ignoring contradictory evidence. EDA should be approached with an open mind, allowing data to speak rather than forcing it into predetermined narratives.

Over-reliance on Means

The mean can be misleading when data is skewed or contains outliers. Always examine medians and full distributions. For GDP data, the mean might suggest average prosperity while hiding extreme inequality.

Ignoring Context

Statistical patterns without domain knowledge can lead to absurd conclusions. A correlation between ice cream sales and drowning deaths doesn’t mean ice cream causes drowning—both increase during summer.

Misleading Visualizations

Poor chart choices, manipulated axes, cherry-picked data ranges, and inappropriate color schemes can distort reality. Always ensure visualizations accurately represent data without exaggeration or omission.

Correlation-Causation Confusion

Perhaps the most dangerous pitfall is inferring causation from correlation. While Gapminder shows strong correlation between GDP and life expectancy, this doesn’t prove that increasing GDP directly causes longer lives. Confounding variables like education, healthcare access, and sanitation play crucial roles.

Neglecting Outliers

While outliers can distort analyses, they often contain valuable information. Rwanda’s drastically low life expectancy in 1992 isn’t noise to be removed—it’s a crucial data point reflecting historical tragedy that demands acknowledgment and investigation.

Analysis Paralysis

EDA is meant to be iterative and exploratory, but endless exploration without moving toward conclusions wastes resources. Setting clear objectives and timelines prevents perpetual analysis without action.

Conclusion

Exploratory Data Analysis, powered by thoughtful visualization, transforms raw data into actionable insights. The Gapminder dataset exemplifies how systematic exploration of metrics like life expectancy, population, and GDP per capita can reveal global development patterns, inequalities, and trends. By understanding key statistical measures—from means and medians to correlations and percentiles—and avoiding common analytical pitfalls, data analysts can extract genuine value from information.

In our data-driven world, EDA skills are not merely technical competencies but essential literacies for making informed decisions. Whether examining global health trends, business performance, or scientific phenomena, the principles remain constant: approach data with curiosity, visualize thoughtfully, measure carefully, and always question assumptions. Through disciplined EDA, we transform data from mere numbers into knowledge that drives understanding and progress.

Subscribe now

Subscribe now

This post draws from the Fast Prototyping of GenAI Apps with Streamlit course by DeepLearning.AI and Snowflake . This course is taught by Chanin Nantasenamat , Sr Developer Advocate also known as the Data Professor.

Why Fast Prototyping Matters

Ideas often lose momentum when they stay in documents or slide decks. In AI, where experimentation is key, the faster you can test and share, the better. Rapid prototyping helps you:

• Validate assumptions: Instead of debating whether an idea works, you can show it.

• Gather feedback early: A working demo sparks more useful conversations than abstract descriptions.

• Iterate quickly: You can refine based on real interactions, not speculation.

• Influence decisions: Stakeholders respond to tangible prototypes more than theoretical plans.

Generative AI adds another layer: with large language models (LLMs), you can build functional prototypes with minimal code. Streamlit provides the interface to make those prototypes usable and shareable.

What is Streamlit?

Streamlit is a Python library that transforms scripts into interactive web apps. Its appeal lies in simplicity:

• Minimal code: A few lines can create buttons, sliders, text inputs, or file uploaders.

• No front-end skills required: You don’t need to know HTML, CSS, or JavaScript.

• Instant sharing: Apps can be deployed on Streamlit Community Cloud or integrated into platforms like Snowflake.

For data scientists, researchers, and AI developers, this means you can focus on logic and models while still delivering polished, interactive demos.

The Prototyping Workflow

The course outlines a practical workflow for building GenAI apps with Streamlit. Here’s a simplified version:

1. Start Small
   Begin with a minimal app—often a chatbot powered by an LLM. The goal is to get something working quickly, not to perfect it.

2. Layer in Prompt Engineering
   Improve the quality of responses by refining prompts. Streamlit makes it easy to expose prompt variations through text boxes or dropdowns, so you can experiment interactively.

3. Add Retrieval-Augmented Generation (RAG)
   Connect your app to external data sources. For example, you might let the chatbot answer questions based on a company’s knowledge base or a dataset stored in Snowflake.

4. Deploy for Feedback
   Push the prototype to Streamlit Community Cloud or Snowflake. Share the link with colleagues or users and gather feedback.

5. Iterate
   Use the feedback to refine prompts, improve data connections, or adjust the interface. Because Streamlit apps are lightweight, iteration cycles are fast.

Example: A Simple Chatbot

Here’s a minimal Streamlit app that connects to an LLM (using OpenAI as an example):

import streamlit as st
import openai

# Set your OpenAI API key
openai.api_key = st.secrets[”OPENAI_API_KEY”]

st.title(”Quick Chatbot Prototype”)

# Initialize chat history
if “messages” not in st.session_state:
    st.session_state.messages = []

user_input = st.text_input(”Ask me anything:”)

if user_input:
    # Append user message to chat history
    st.session_state.messages.append({”role”: “user”, “content”: user_input})

    # Call OpenAI ChatCompletion with the full conversation history to maintain context
    response = openai.ChatCompletion.create(
        model=”gpt-3.5-turbo”,
        messages=st.session_state.messages
    )
    
    answer = response[”choices”][0][”message”][”content”]
    
    # Append assistant’s response to chat history
    st.session_state.messages.append({”role”: “assistant”, “content”: answer})
    
    # Display the assistant’s response
    st.write(answer)

This script:

• Creates a text input box.

• Sends the input to an LLM.

• Displays the response.

It’s only a few lines of code, but it produces a working chatbot you can share.

Adding Prompt Engineering

Prompt engineering is about shaping the model’s behavior. With Streamlit, you can expose prompt templates as editable fields:

import streamlit as st
import openai

# Set OpenAI API key securely
openai.api_key = st.secrets[”OPENAI_API_KEY”]

prompt_template = st.text_area(
    “Prompt template:”,
    “You are a helpful assistant. Answer clearly and concisely.\n\nUser: {question}\nAssistant:”
)

user_input = st.text_input(”Ask me anything:”)

if user_input:
    prompt = prompt_template.format(question=user_input)
    response = openai.Completion.create(
        model=”text-davinci-003”,
        prompt=prompt,
        max_tokens=200
    )
    st.write(response[”choices”][0][”text”])

Now you can experiment with different instructions without changing the code.

Adding RAG (Retrieval-Augmented Generation)

RAG combines LLMs with external data. For example, you might let the chatbot answer based on a set of documents. A simplified version looks like this:

import streamlit as st
import openai

openai.api_key = st.secrets[”OPENAI_API_KEY”]

prompt_template = st.text_area(
    “Prompt template:”,
    “You are a helpful assistant. Answer clearly and concisely.\n\nUser: {question}\nAssistant:”
)

user_input = st.text_input(”Ask me anything:”)

if user_input:
    prompt = prompt_template.format(question=user_input)
    response = openai.Completion.create(
        model=”text-davinci-003”,
        prompt=prompt,
        max_tokens=200
    )
    st.write(response[”choices”][0][”text”])

This setup allows the chatbot to ground its answers in your own dataset, making it more useful for specific domains.

Deployment

When building a prototype, keep things simple and focus on the main interaction—don’t add unnecessary features. Make it easy for users to adjust prompts, settings, or data inputs. Share your work early and improve it quickly through feedback. If your prototype handles sensitive data, make sure it runs in a secure environment.

Once your prototype works locally, you can deploy it:

• Streamlit Community Cloud: Free and simple for sharing demos.

• Snowflake + Streamlit: For secure, production-ready environments with enterprise data.

Deployment is as simple as pushing your code to GitHub and linking it to Streamlit Cloud.

Wrapping Up

Streamlit doesn’t replace production systems, but it gives you a powerful way to explore, test, and communicate ideas. In the fast-moving world of generative AI, that speed of exploration is often the difference between leading and lagging.

Rapid prototyping with Streamlit is less about building polished products and more about accelerating learning. By lowering the barrier to creating interactive GenAI apps, it allows individuals and teams to validate ideas quickly, gather feedback, and refine direction.

In practice, this means fewer stalled discussions and more tangible progress. Whether you’re experimenting with a chatbot, a summarizer, or a data assistant, the workflow is the same: start small, iterate fast, and share early.

Share ValueCurve

Subscribe now

The purpose of this post is to provide readers with a mental map for any ML project. Let’s dive in.

1. Data Collection

The process begins with gathering relevant data from various sources. These might include databases, logs, CSV files, APIs, and sensor feeds. It’s crucial to define what is needed before pulling in every field. Objectives and data schema should be sketched out early to save time in later stages. Privacy rules and security policies must be respected during record ingestion. Each dataset should be tagged with metadata that captures when, where, and how it arrived. This context supports traceability and reproducibility in analysis and modeling. A solid foundation of clean, well-understood inputs reduces surprises down the road.

2. Feature Engineering

Raw attributes rarely arrive in the shape and scale a model requires. Feature engineering transforms these inputs into meaningful variables. Numeric fields might be normalized, categorical ones encoded, or parts of timestamps extracted. Domain expertise helps craft interaction terms or rolling averages for time-series data. For text tasks, tokenization or embeddings convert words into numbers. Techniques like PCA or feature selection help manage high dimensionality. Each crafted feature should carry a clear rationale about its role in prediction. Well-engineered features often boost performance more than complex algorithms.

3. Model Training

With features in place, a model is trained to learn underlying patterns. The process starts by selecting an algorithm family—linear models, decision trees, or neural networks—based on the problem. Data is split into training and validation sets to catch overfitting before it sneaks in. Hyperparameter tuning is automated with grid search, random search, or Bayesian methods. Training metrics like loss curves and convergence behavior are monitored. Each experiment’s configuration and results are tracked with an experiment management tool. Automating repeatable training pipelines prevents manual errors and ensures reproducibility. A robust training loop lays the groundwork for reliable predictions.

4. Evaluation

Evaluation measures how well a model performs on unseen data. Metrics should align with real-world goals—accuracy, precision/recall, RMSE, AUC, or custom business KPIs. A hold-out test set that never touched training or tuning is reserved to avoid leakage. Performance is visualized with confusion matrices or ROC curves to spot strengths and weaknesses. Error cases are drilled into to understand model blind spots. Bias is checked by comparing results across different subgroups. Findings are documented clearly to guide stakeholders on strengths and limitations. Thorough validation builds confidence before production rollout.

5. Deployment

Deployment turns a validated model into a usable service or batch task. The model is containerized with Docker or packaged for serverless platforms. Inference logic is wrapped in a simple API or integrated into an existing data pipeline. Model artifacts and API definitions are version-controlled to track changes over time. Deployment pipelines are automated to eliminate manual hand-offs and reduce errors. Endpoints are load-tested to verify they meet latency and throughput requirements. Fallback logic or circuit breakers are included to handle unexpected failures gracefully. Smooth, repeatable deployment processes make going live painless and predictable.

6. Monitoring

Once in production, a model enters a dynamic environment that can drift over time. Input feature distributions are monitored to catch covariate drift early. Performance metrics are tracked against expected baselines or business KPIs continuously. Logs are instrumented for latency, error rates, and resource usage. Alerts are set to trigger when metrics deviate beyond safe thresholds. Diagnostic dashboards are used to spot anomalies at a glance. Regular health checks ensure the service remains stable under varying loads.

7. Maintenance

Maintenance keeps a model effective as data and requirements evolve. Over time, input distributions shift, feature relevance changes, and new business rules emerge. A proactive maintenance plan ensures the pipeline doesn’t silently degrade.

A retraining cadence is established based on data volume and observed drift. Every model artifact—training code, feature transformations, and weights—is versioned to allow rollback or comparison of historic performance.

Tests are automated to validate feature pipelines and inference logic after updates. New data sources or labels are incorporated when business needs shift, then their impact is validated through the evaluation suite. Obsolete features are rotated out and their effect on performance monitored to prevent bloat.

Wrapping Up

A robust machine learning pipeline weaves together data collection, feature engineering, model training, evaluation, deployment, monitoring, and maintenance. Each stage builds on the last to create a transparent, reproducible path from raw inputs to reliable predictions. Skipping any phase invites hidden bias, untraceable errors, and brittle systems.

When this end-to-end flow is followed; the right data is captured with clear ownership. That data is distilled into features that carry meaningful signals. Models are trained under controlled conditions and validated on truly unseen cases. Deployment is handled with repeatable pipelines, version control, and safety nets. Monitoring is continuous, adapting to real-world drift rather than hoping nothing breaks.

With this structure in place, Engineers can iterate on features or algorithms the team can focus on building next-generation capabilities. More importantly, Stakeholders can see clear metrics that connect model outputs to business impact.

Share ValueCurve

Subscribe now

Core Plotting and Customization

Keith Galli, a MIT graduate has published over 100 videos on Computer Science, Programming & Data Analysis. Keith created Python plotting crash course using matplotlib for beginners, few years ago. This course begins with the fundamentals of plotting before moving into customization.

• Setup: Import the libraries: matplotlib.pyplot as plt, numpy as np, and pandas as pd.

• Basic Line Graph: Create a simple line graph with plt.plot(x, y) and display it using plt.show().

• Labels and Titles: Add context with plt.title(), plt.xlabel(), and plt.ylabel().

• Tick Marks: Adjust axis tick values with plt.xticks() and plt.yticks() for readability.

• Legends: Use plt.legend() to label multiple datasets; it will auto-place the legend.

• Line Styling: Control appearance with parameters such as color, linewidth, marker, markersize, and linestyle.

• Shorthand Notation: Combine style settings quickly, e.g. ‘b.-’ for a blue line with dots.

• Font Customization: Pass a fontdict to titles or labels to change size, family, and weight.

• Figure Size and Resolution: Define figure dimensions with plt.figure(figsize=(w, h), dpi=300).

• Saving a Plot: Export graphics with plt.savefig(’plot.png’, dpi=300).

Variety of Plot Types

The course covers multiple visualization methods beyond line graphs:

• Line Graphs — e.g., gas price trends.

• Bar Charts — plt.bar(), with bar patterns via set_hatch().

• Histograms — plt.hist() to show data distributions, with adjustable bins.

• Pie Charts — plt.pie(), including labels, percentages (autopct), custom colors, and slice separation (explode).

• Box and Whiskers Plots — plt.boxplot() for comparing distributions, with options to style boxes.

Real-World Application with Pandas

Integration with Pandas is a key feature:

• Loading Data: Read CSVs with pd.read_csv().

• Example Datasets:

  • Gas Prices — country-level prices over time.

  • FIFA Player Stats — player attributes from the FIFA game.

• Data Selection: Extract subsets before plotting, such as gas[’USA’] or filtering with FIFA.loc[FIFA[’Club’] == ‘FC Barcelona’].

Teaching Approach & Takeaway

The course keeps a practical focus:

• Documentation First: Suggests to use the official Matplotlib docs for functions and parameters.

• Problem Solving: Search Google or Stack Overflow for specific issues, as is common in real development. This can now be augmented with AI tools.

This course covers the essentials of Matplotlib: how to create plots, customize them, work with multiple chart types, and apply them to real datasets through Pandas. It’s a practical introduction for anyone learning data visualization in Python. The datasets required to complete the matplotlib course can be downloaded from github . Additionally, you can also download cheatsheets for reference from the official site.

Share ValueCurve

Classes and Objects

• Object: Everything in Python is an object, and every object has a type that defines its data and behaviors.

• Class: A class acts as a blueprint for creating objects.

• Instance (Object): A specific creation based on that class blueprint.

Defining a Class

You define a class using the class keyword, followed by the class name. In Python 3, it’s common to write class Coordinate: instead of class Coordinate(object): because all classes inherit from object by default.

Attributes

• Data attributes (instance variables): Values stored per object, typically defined in __init__.

• Methods: Functions defined inside a class that operate on its instance data.

__init__ Method and self

• Constructor (__init__): Called automatically when a new instance is created; initializes the object’s attributes.

• self parameter: Refers to the instance on which a method is called. It is passed implicitly; self is the convention, not a Python keyword.

Data Attributes

Defined inside __init__ with self (e.g., self.x = xval). These persist for the object’s lifetime.

Dunder Methods and Operator Overloading

• Special Methods (Dunder Methods): Begin and end with double underscores, such as __init__, __str__, __add__.

• __str__: Defines a human-readable string (used in print() calls). Falls back to __repr__ if not defined.

• Operator Overloading: By defining methods like __add__, __eq__, and __mul__, you control how operators work on your custom objects.

Inheritance

• Concept: A child (subclass) can inherit attributes and methods from a parent (superclass), e.g., class Cat(Animal):.

• Overriding: A child can replace a parent’s method with its own definition.

• Extending: A child can add new attributes/methods.

• Method Resolution Order (MRO): Python looks for attributes in the current class, then parent(s), following a consistent MRO for multiple inheritance.

Class Variables

• Variables defined inside the class but outside methods are shared by all instances.

• Be careful: assigning to self.var in an instance will create/override an instance attribute, shadowing the class variable.

  Thanks for reading ValueCurve! This post is public so feel free to share it.

  Share

Python Code Example: Defining a Coordinate Class

This code block demonstrates how to define a class, create a constructor (__init__), define data attributes, and implement a procedural method (distance). It also shows the power of the __str__ method.

import math

class Coordinate:
    # Class Variables (shared by all instances)
    dimension = 2
    instance_count = 0

    def __init__(self, x_val, y_val):
        # Instance Variables
        self.x = x_val
        self.y = y_val
        Coordinate.instance_count += 1

    def distance(self, other_coord):
        “”“Calculates the Euclidean distance to another Coordinate.”“”
        x_diff = self.x - other_coord.x
        y_diff = self.y - other_coord.y
        return math.sqrt(x_diff**2 + y_diff**2)

    def __str__(self):
        “”“Defines a human-readable string representation.”“”
        return f”<{self.x}, {self.y}>”

    def __add__(self, other_coord):
        “”“Overloads ‘+’ operator to add coordinates (vector addition).”“”
        new_x = self.x + other_coord.x
        new_y = self.y + other_coord.y
        return Coordinate(new_x, new_y)

# --- Creating Objects ---
point1 = Coordinate(3, 4)
point2 = Coordinate(0, 0)
point3 = Coordinate(10, 5)

# --- Accessing Attributes and Methods ---
print(f”Point 1 X-attribute: {point1.x}”)
print(f”Distance between {point1} and {point2}: {point1.distance(point2):.2f}”)
print(f”Point 3 representation: {point3}”)

# --- Operator Overloading ---
point_sum = point1 + point3
print(f”Point Sum (point1 + point3): {point_sum}”)

# --- Class Variables ---
print(f”Total Coordinate objects created: {Coordinate.instance_count}”)
print(f”Dimension (class variable): {point1.dimension}”)

Summary and Best Practices

OOP provides the tools to manage large, dynamic projects by modeling complexity with clarity, structure, and consistency. Mastering these fundamentals is the key to advancing beyond basic scripting into professional software development.

The central mechanisms are:

• Instantiation: The __init__ method and the self parameter construct new objects and bind instance-specific data.

• Specialization: Inheritance allows creation of new classes that extend or override the behavior of their parents, promoting code reuse and clarifying relationships.

• Integration: Dunder methods like __str__ and __add__ let custom objects integrate naturally with Python’s built-in functions and operators, producing more readable and “Pythonic” code.

Best practices include:

• Designing classes around a clear, single responsibility to keep them maintainable and understandable.

• Using __str__ for human-readable output and __repr__ for accurate, developer-facing representations.

• Managing class variables carefully, recognizing when values should be shared at the class level versus kept per instance.

• Favoring composition (building complex objects from simpler ones) when it improves clarity, rather than relying heavily on inheritance hierarchies.

• Applying operator overloading selectively, only when it genuinely improves code clarity and aligns with intuitive object behavior.

OOP is powerful in Python but should be used thoughtfully, alongside procedural and functional styles, to build flexible, scalable, and maintainable systems.

Subscribe now

Share ValueCurve

Key Characteristics and Operations

Mutability: The most significant feature of a list is its mutability. Unlike immutable objects such as strings and tuples, lists can be changed in memory after they are created. This means you can alter a list object itself without creating a new copy. This is a key difference from tuples, where reassigning a variable to a new tuple creates a new object in memory, leaving the original unchanged. Because of their mutable nature, lists are highly efficient for managing dynamic data, like a list of employees or students, as changes don’t require creating entirely new copies of large data structures.

Modifying Elements: You can change an element at a specific index using assignment syntax, such as L[index] = 5.

Adding Elements:

• append(): Adds a single item to the end of a list, mutating the original list. Returns None.

• extend(): Adds all elements from another list to the end of the original.

Removing Elements:

• remove(element): Removes the first occurrence of a specified value.

• del L[index]: Deletes the element at a specific index.

• pop(): Removes and returns the last element.

• clear(): Removes all elements, keeping the same list object in memory.

Sorting and Reversing:

• L.sort(): Sorts the list in place, returning None.

• L.reverse(): Reverses the list in place.

• sorted(L): Returns a new, sorted copy without altering the original.

Iteration:

• Iterate directly with for e in L: (more Pythonic).

• Use for i in range(len(L)): if you need to mutate elements by index.

• Mutating a list while iterating directly can cause skipped elements; safer: iterate over a copy.

Copying (Cloning):

• Shallow copy: L_copy = L[:] (top-level only).

• For independent nested structures, use copy.deepcopy(L).

List Comprehension:

• Pattern: [expression for item in sequence if condition].

• A concise way to replace explicit loop + append structures.

Summary and Best Practices

In summary, the Python list is a versatile and indispensable collection. To use it effectively and avoid common pitfalls, keep these core principles in mind:

• Embrace Mutability: Use in-place methods like append(), extend(), and sort() for efficient data alteration, but be mindful that these methods always mutate the original list.

• Beware of Aliasing: Use shallow copy techniques such as L[:], list(L), or L.copy() when you need an independent copy of a list’s top-level structure. This prevents unintentional modifications to the original list. If the list contains nested mutable structures (e.g., lists of lists), use copy.deepcopy(L) when you need a fully independent copy of both top-level and nested objects.

• Use List Comprehensions: Adopt the [expression for item in sequence if condition]pattern to write cleaner, faster, and more readable code when generating new lists.

• Prefer Safe Mutations During Iteration: Mutating a list while iterating can cause skipped elements. Iterate by index or over a copy for safety.

• Understand Sorting Options: Use list.sort() when you want to sort in place for performance. Use sorted(list) when you need a new sorted copy without affecting the original.

  Share ValueCurve

Subscribe now

The need for dictionaries arises from the limitations of using lists for complex data storage. Let’s consider this with a student grades example:

• Using parallel lists (one for names, one for grades) is cumbersome. To find a student’s grade, you first have to find their index in the names list and then use that same index to access the grades list. This process becomes increasingly messy as more data (like quiz or problem set scores) is added, requiring more parallel lists that must be kept in sync.

• Using a “master list” of nested lists also proves to be complex and hard to read, requiring nested loops to search for specific information.

Dictionaries solve this by providing a direct mapping from a key (e.g., a student’s name) to a value (e.g., their grade or a more complex data structure containing all their information). An entry in a dictionary is a key-value pair.

Creating, Accessing, and Modifying Dictionaries

Creation

• Dictionaries are created using curly braces {}. An empty dictionary is created with d = {}.

• Entries are defined as key: value, and multiple entries are separated by commas. For example, a dictionary mapping names to grades could be created like this: grades = {’Ana’: ‘B’, ‘Matt’: ‘A’, ‘John’: ‘B’}.

Accessing and Mutability

• Dictionaries are mutable objects, meaning they can be changed after they are created.

• To look up a value, you use the key as an index in square brackets, similar to list indexing (e.g., grades[’John’] would return ‘B’).

• If you try to access a key that does not exist in the dictionary, Python will raise a KeyError exception.

• You cannot look up a key by its value directly, as values can be duplicated. You would need to write a loop to perform this reverse lookup.

Operations

• Adding or Modifying Entries: The same syntax is used for both adding a new entry and changing an existing one. If the key does not exist, a new entry is created. If it does exist, its value is overwritten. For example, grades[’Grace’] = ‘A’ adds Grace, and grades[’Grace’] = ‘C’ later changes her grade.

• Deleting Entries: The del keyword is used to remove an entry, such as del grades[’Ana’]. This mutates the original dictionary.

• Checking for a Key: The in operator checks for the existence of a key in a dictionary (e.g., ‘John’ in gradesreturns True). It is important to note that the in operator only checks keys, not values.

• Copying: Since dictionaries are mutable, simple assignment (d2 = d1) creates an alias (another name for the same object). To create a shallow copy, you must use the .copy() method (d2 = d1.copy()).

  ---

Iteration

While recent versions of Python guarantee insertion order for dictionaries, it is best practice to write code assuming they are unordered sequences for robustness and compatibility with older versions. There are three main ways to iterate over a dictionary:

• .keys(): Returns an iterable sequence of all the keys in the dictionary.

• .values(): Returns an iterable sequence of all the values.

• .items(): This is often the most effective method, as it returns an iterable of key-value pairs as tuples. This allows you to access both the key and value simultaneously in a loop, for example: for k, v in my_dict.items():.

Restrictions on Keys and Values

• Dictionary values can be of any type, including other mutable objects like lists or other dictionaries. Values can also be duplicated.

• Dictionary keys have two main restrictions:

  1. They must be unique. You cannot have two entries with the same key.

  2. They must be immutable (technically, “hashable”). This means keys can be types like integers, floats, strings, tuples, and booleans, but cannot be lists or other dictionaries.

Performance and Implementation: Hashing

Following is a detailed explanation of why dictionaries offer superior performance for lookups compared to lists and why keys must be immutable :

• Average-Case constant time lookup: Dictionaries are implemented using a hash table. A hash function is run on a key, which converts it into an integer. This integer is then used as a direct index to find the location of the corresponding value in the hash table (which is structured like a list in memory). Because this lookup is based on a direct calculation rather than a sequential search, accessing an item in a dictionary takes constant time (Θ(1)) on average. This provides a massive performance advantage over searching a list, which takes linear time (Θ(n)).

• Collisions and worst-case performance: A “collision” occurs when two different keys produce the same hash value, mapping them to the same location or “bucket” in the hash table. These collisions are handled by storing the colliding entries together, often in a list-like structure within that bucket. A good hash function distributes keys uniformly, minimizing collisions. However, in the worst-case scenario, if all keys collide into the same bucket, looking up an item degrades to a linear search through that bucket, resulting in linear time (theta(n)) performance.

• Why keys must be immutable: The hashing mechanism requires that a key always produces the same hash value. If a key were a mutable object like a list, its contents could change after it was stored in the dictionary. This change would alter its hash value, making it impossible for the dictionary to find the original location where the value was stored.

Use Cases

1. Storing Structured Data: Dictionaries are excellent for representing complex, structured data. The student information example was revisited to show how a nested dictionary (grades[’Ana’][’ps’]) provides a much cleaner and more efficient way to store and retrieve data compared to nested lists.

2. Counting Frequencies: A common pattern is to create a “frequency dictionary” to count occurrences of items. An example shows how to build a dictionary that maps each word in a song’s lyrics to the number of times it appears.

3. Memoization: Dictionaries are crucial for an optimization technique called memoization, which is used to speed up expensive computations, particularly in recursive functions. The Fibonacci example demonstrates this by storing previously calculated fib(n) values in a dictionary. Before computing a value, the function checks if the result is already in the dictionary, avoiding redundant calculations. This improved the efficiency of the Fibonacci function from exponential time to a much faster one, reducing millions of function calls to just a few dozen.

Summary and Best Practices

Python dictionaries are the core tool for managing unordered, dynamic data that requires fast, key-based lookup. They fundamentally map unique, hashable keys to corresponding values.

The dictionary’s defining characteristic is its reliance on hash tables, which allows for item retrieval in average-case constant time (Θ(1)), offering a massive performance advantage over sequential list searching.

Share ValueCurve

Subscribe now

Learning to program is a skill that requires consistent practice. It is not just about understanding concepts but also about developing problem-solving ability—the process of translating a problem described in English into a computational solution. Following concepts should serve as a memory aid for the beginners in Python.

Objects, Types, and Variables

• Objects and Types: In Python, everything is an object, from numbers to functions. Each object has a type that defines what operations can be performed on it. You can use type()to check an object’s type.

• Scalar Data Types:

  • int: Whole numbers like 5, 0, -100.

  • float: Real numbers with a decimal point, such as 3.27 or 2.0. Because of binary representation, floating-point operations may introduce tiny rounding errors.

  • bool: Truth values—True or False (case-sensitive).

  • NoneType: Has only one value, None, used to represent the absence of a value.

• Variables and Expressions:

  • Variables are names bound to objects. The assignment operator = binds the name on the left to the evaluated value on the right.

  • Example: area = pi * radius**2 first evaluates the expression and then assigns it to area.

  • Variables can be rebound to new values. For instance, x = x + 1 calculates x + 1 and rebinds x to this result.

• Operators:

  • Arithmetic: +, -, *, **, / (float division), // (integer division), % (modulo).

  • Comparison: ==, !=, >, < evaluate to True or False.

  • Logical: and, or, not.

Input, Output, and Control Flow

• Input and Output:

  • print() displays output.

  • input() reads user input as a string. Casting is often required, e.g., age = int(input(”Enter age: “)).

  • f-strings provide formatted string interpolation: print(f”Area = {area}”).

• Control Flow:

  • Branching: if, elif, else allow decision-making. Indentation defines code blocks.

  • Loops:

    • while: Repeats while a condition is True. Be careful to avoid infinite loops.

    • for: Iterates over sequences. Example: for i in range(5):.

Functions

• Defining and Calling: Declared with def and executed when called.

• Parameters and Return Values: Accept inputs and return results with return. Functions without a return default to None.

• First-Class Objects: Functions can be assigned to variables, passed as arguments, or returned from other functions.

• Scope: Each function has its own local scope.

  • Local variables are destroyed after the function finishes.

  • Functions can read variables from outer scopes but cannot modify them unless declared with global or nonlocal.

• Loop Control:

  • break: Exit the loop early.

  • continue: Skip to the next iteration.

  • pass: Placeholder that does nothing (useful for code stubs).

Common Data Structures

• Strings (str): Immutable, ordered sequence of characters.

  • Created with quotes: “hello”.

  • Operations: concatenation (+), repetition (*), length (len()), indexing, slicing (s[1:3]).

• Tuples (tuple): Immutable, ordered collection of objects.

  • Example: (2, ‘MIT’, 3).

  • Support indexing, slicing, and unpacking (x, y = y, x).

  • Often used to return multiple values from a function.

• Lists (list): Mutable, ordered collection.

  • Example: [2, ‘a’, 4].

  • Elements can be modified: L[0] = 5.

  • Common methods:

    • L.append(item) – add item at end

    • L.extend(other) – append all items from another list

    • L.sort() – sort list in place

    • L.reverse() – reverse in place

    • L.remove(item) – remove first matching element

  • Assigning newList = oldList creates an alias, not a copy. For a copy: newList = oldList[:].

• Dictionaries (dict): Mutable collection of key-value pairs.

  • Example: grades = {’Ana’: ‘B’, ‘John’: ‘A’}.

  • Keys must be immutable types (str, int, tuple).

  • Access: grades[’Ana’].

  • Add or modify: grades[’Grace’] = ‘A’.

  • Iteration: .keys(), .values(), .items().

  • Safe access: grades.get(’Ana’, ‘Not Found’).

Exception and Error Handling

Python provides robust mechanisms to handle errors using try, except, and finally blocks. While not essential at the very beginning, learning exceptions is important once you start writing programs that deal with unpredictable input, files, or external data.

Summary & Best Practices

Python is an imperative, object-oriented language in which every entity is treated as an object. Its core components include variables, operators, control flow mechanisms, data structures, and functions.Mutability plays a key role: lists and dictionaries are mutable, while strings and tuples are immutable. Functions support abstraction, modularity, and reuse, with built-in features like default and keyword arguments.

Clean Python code is easier to read and maintain when variable names clearly reflect their purpose. For loops are typically more readable than while loops, unless the loop depends on a specific condition and f-strings simplify string formatting by embedding expressions directly. Mutable objects like lists and dictionaries can lead to unintended side effects when shared across functions or scopes, so they require careful handling.

Deeply nested logic can obscure intent; breaking it into smaller functions improves clarity and reuse. User input should be converted and validated to prevent errors. Comments are useful when necessary, but well-structured code and meaningful names often make them redundant. By combining a solid grasp of the language’s core constructs with disciplined coding practices, developers can build systems that are both elegant and resilient.

Share ValueCurve

Tuples as Data Structures

Tuples are a fundamental Python data structure—a way of organizing data with a focus on immutability and reliability. Unlike lists, tuples guarantee their contents won’t change, which makes them suitable in contexts such as dictionary keys, fixed records, or multiple return values

Share

Tuples: Immutable Sequences

A tuple is an ordered collection of values, typically enclosed in parentheses `()`. Unlike lists, tuples cannot be modified once defined.

coordinates = (10, 20)
# Trying to change an element raises an error:
# coordinates[0] = 99   # TypeError: ‘tuple’ object does not support item assignment

This immutability makes tuples ideal for storing fixed collections of related data where consistency matters.

Packing: Collecting Values into a Tuple

Packing is a feature where multiple values are grouped together into a tuple automatically.

point = 5, 7
print(point)   # Output: (5, 7)

This allows for easy grouping of values without explicit tuple syntax.

Unpacking: Assigning Tuple Elements

Unpacking takes a tuple and assigns its elements to individual variables in one step.

x, y = point
print(x, y)   # Output: 5 7

Unpacking is useful when functions return multiple values or when processing tuple-based data.

Subscribe now

`zip`: Combining Iterables

The `zip()` function pairs elements from multiple iterables into tuples, providing a combined view of the data.

names = [”Alice”, “Bob”, “Charlie”]
scores = [85, 90, 95]

zipped = zip(names, scores)
print(list(zipped))  
# Output: [(’Alice’, 85), (’Bob’, 90), (’Charlie’, 95)]

This is helpful for parallel iteration over multiple sequences.

`enumerate`: Indexed Iteration

When you want both the index and the value from a sequence, `enumerate()` produces tuples of `(index, value)` pairs.

fruits = [”apple”, “banana”, “cherry”]

for index, fruit in enumerate(fruits, start=1):
    print(index, fruit)

# Output:
# 1 apple
# 2 banana
# 3 cherry

Sorting with a Key

Tuples can represent structured data, such as `(name, score)` pairs. When sorting collections of tuples, a sort key can determine the criterion.

students = [(”Alice”, 85), (”Bob”, 90), (”Charlie”, 78)]

sorted_students = sorted(students, key=lambda student: student[1])
print(sorted_students)
# Output: [(’Charlie’, 78), (’Alice’, 85), (’Bob’, 90)]

This sorts the list based on the score (the second tuple element).

Combined with Python’s powerful features like packing, unpacking, `zip`, and `enumerate`, tuples provide a simple but robust way to handle fixed collections of data efficiently and safely.

Stay Connected

Subscribe now

The Challenge

The task is to write a program that, given two lowercase phrases (with spaces and punctuation removed), determines whether they are anagrams. For instance, listen and silent are anagrams, while hello and world are not. Though often used as a beginner coding exercise, this challenge has real-world relevance in systems that require fast, reliable string comparison—such as search engines, data deduplication, and secure identifier matching.

Counting Characters

Two strings are anagrams if and only if they contain the same characters with the same frequency. For example, both listen and silent contain one l, one i, one s, one t, one e, and one n. If the character counts match for all letters, the strings are anagrams. This principle forms the basis of the program’s logic: rather than comparing order, it compares composition.

Algorithm in Three Steps

1. Length Check: Return false immediately if the lengths differ, as they cannot be anagrams.

2. Frequency Counting: Iterate over both strings, counting characters using dictionaries.

3. Comparison: Compare the two dictionaries; if they match, print "true", else "false".

This approach ensures efficiency and clarity, especially when scaled to multiple test cases.

Python Code

def solve_anagrams():
    try:
        print("Anagram Verifier")
        print("----------------")
        print("Enter a number between 1 and 20 for anagram pairs to check")
        t = int(input())  # Number of anagram pairs to check 
        if not (1 <= t <= 20):
            return

        for _ in range(t):
            line = input().split()
            s1, s2 = line[0], line[1]

            # Step 1: Length check
            if len(s1) != len(s2):
                print("false")
                continue

            # Step 2: Build character frequency dictionaries
            count1, count2 = {}, {}
            for char in s1:
                count1[char] = count1.get(char, 0) + 1
            for char in s2:
                count2[char] = count2.get(char, 0) + 1

            # Step 3: Compare dictionaries
            print("true" if count1 == count2 else "false")
    except (IOError, ValueError):
        return

solve_anagrams()

Practical Applications

This kind of string comparison has a wide range of uses. In security, anagram logic can help obscure data while preserving its verifiability. In games and puzzles, it’s a core mechanic for word challenges. In natural language processing, it helps in analyzing lexical variation and stylistic patterns. And in systems design, scrambling identifiers using anagram principles can enhance privacy without losing uniqueness.

While sorting both strings is also a valid method for checking anagrams, its performance degrades as input size grows. Sorting requires approximately n × log n steps (O(n log n) for those familiar with the Big O notation ), where n is the length of the string. Doubling the string length results in more than double the processing time. In contrast, counting character frequencies and comparing them takes only O(n) time, since it involves a single pass through each string. This linear-time approach is significantly faster and more scalable, making it the preferred method for high-performance applications.

To make the program more robust, it could be extended to handle cases such as Unicode or accented characters, which are common in multilingual datasets. From a performance standpoint, counting characters is generally more efficient than sorting, especially when dealing with long strings or large volumes of comparisons. For a cleaner implementation, Python’s collections.counter class offers a concise and powerful way to perform frequency-based comparisons with minimal code.

Share ValueCurve

1. Installing Visual Studio Code

Visual Studio Code (VSCode) is a powerful, free code editor that's essential for development.

Download and Install:

1. Visit code.visualstudio.com

2. Click "Download for Mac"

3. Open the downloaded .zip file

4. Drag the Visual Studio Code app to your Applications folder

5. Launch VSCode from Applications or Spotlight

First Launch Setup:

• VSCode may ask for permissions to access folders - click "Allow"

• Install the Python extension by Microsoft from the Extensions panel (Cmd+Shift+X)

  

Subscribe now

2. Installing and Configuring Python

Python is the most popular language for AI development, offering extensive libraries and frameworks.

Install Python using Homebrew:

1. Open Terminal (press Cmd+Space, type "Terminal", press Enter)

2. First, install Homebrew (if not already installed):

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

3. Install Python:

brew install python

4. Verify installation:

python3 --version
pip3 --version

Configure Python Environment:

1. In Terminal, create a virtual environment for your projects:

python3 -m venv ~/ai-projects

2. Activate the environment:

source ~/ai-projects/bin/activate

3. Install essential packages:

pip install jupyter pandas numpy matplotlib requests

VSCode Python Setup:

1. Open VSCode and press Cmd+Shift+P

2. Type "Python: Select Interpreter"

3. Choose the interpreter from your virtual environment (~/ai-projects/bin/python)

Configure Shell Profile: In Terminal, add the virtual environment path to your shell profile for easy activation:

echo 'alias activate-ai="source ~/ai-projects/bin/activate"' >> ~/.zshrc
source ~/.zshrc

Now you can simply type activate-ai to activate your environment.

3. Installing and Configuring Claude CLI

Claude CLI enables direct interaction with Claude AI from your terminal, streamlining AI-assisted development.

Installation:

1. In Terminal, install Node.js (required for Claude CLI):

brew install node

2. Install Claude CLI globally:

npm install -g @anthropic-ai/claude-cli

Configuration:

1. Get your API key from console.anthropic.com

2. In Terminal, configure Claude CLI:

claude configure

3. Enter your API key when prompted

4. Test the installation:

claude --version

Basic Usage:

• Start a conversation: claude chat

• Get help with code: claude "explain this Python function" < myfile.py

• Generate code: claude "create a Python script to read CSV files"

  Share

4. Configuring VSCode with GitHub

GitHub integration enables version control and collaboration, essential skills for any developer.

Install Git: In Terminal, run:

brew install git

Configure Git: Still in Terminal:

git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"

GitHub Authentication:

1. In Terminal, install GitHub CLI:

brew install gh

2. Authenticate with GitHub:

gh auth login

3. Follow the prompts to authenticate via browser

VSCode GitHub Setup:

1. Install the "GitHub Pull Requests and Issues" extension

2. Sign in to GitHub when prompted in VSCode

3. Open the Source Control panel (Cmd+Shift+G)

Creating Your First Repository:

1. Create a new folder for your project

2. Open it in VSCode: File > Open Folder

3. Initialize Git using VSCode's integrated terminal (`Ctrl+``):

git init

4. Create a proper .gitignore file in the VSCode terminal:

# Create .gitignore with common exclusionscat > .gitignore << EOF# Virtual environmentsvenv/env/.env# Python__pycache__/*.pyc*.pyo.Python# IDEs.vscode/settings.json.DS_Store# API keys and secrets.env.localconfig/secrets.json# Jupyter Notebooks.ipynb_checkpoints/EOF

5. Create a README file and make your first commit

6. Push to GitHub using the Source Control panel

Environment Variables Setup: For secure API key management, create a .env file in the VSCode terminal (already in .gitignore):

# Create .env file for secrets
echo "ANTHROPIC_API_KEY=your_api_key_here" > .env

Next Steps

This environment is optimized for AI development. The combination of VSCode's editing capabilities, Python's AI libraries, Claude CLI's assistance, and GitHub's collaboration tools provides everything needed to begin your AI fluency journey.

Remember to activate your Python virtual environment (activate-ai or source ~/ai-projects/bin/activate) each time you start a new terminal session for development work. Always keep sensitive information like API keys in .env files that are excluded from version control.

Leave a comment

Subscribe now

As AI solutions are being increasingly used to communicate, create, learn, and solve problems, the 4Ds framework is designed to help users engage with AI responsibly across all contexts and applications.

Understanding AI Engagement Modes

To understand how this framework applies across different use cases, it's helpful to examine how people currently engage with AI systems. Anthropic classifies existing AI solutions into three categories based on engagement patterns:

Automation Mode: The AI completes specific tasks based on user instructions and works in automation mode.

Augmentation Mode: AI is used as a brainstorming partner. User and AI collaborate as creative thinking and task execution partners.

Agency Mode: Users configure AI to work independently on their behalf as an agent, establishing its knowledge and behavior patterns rather than just giving it specific tasks.

The 4Ds Framework

Regardless of which mode of AI engagement you choose, the 4Ds Framework provides essential competencies for responsible and effective collaboration. Each competency builds upon the others to create comprehensive AI fluency.

1. Delegation

Delegation involves making thoughtful decisions about what work to do independently, what to do collaboratively with AI, and what to let AI handle autonomously, along with determining how to distribute these tasks effectively. This process begins with problem awareness, which means clearly understanding goals and the nature of the work before involving AI systems. Platform awareness involves understanding the capabilities and limitations of different AI systems to make informed choices about which tools to employ. Task delegation represents the strategic process of thoughtfully distributing work between humans and AI to leverage the unique strengths of each participant. Effective delegation requires both domain expertise and a comprehensive understanding of AI capabilities to create optimal partnerships. The ultimate goal is not to automate everything, but rather to create the most effective human-AI partnership for any given task or objective, ensuring that each participant contributes what they do best to achieve superior outcomes.

2. Description

Effective AI communication centers on creating a productive collaborative environment through three key types of description. Product description involves clearly defining desired outputs, format, target audience, and style preferences. Process description guides how the AI should approach requests, which can be as crucial as specifying the end goal itself. Performance description defines behavioral expectations, such as whether responses should be concise or detailed, challenging or supportive. Rather than treating AI systems as databases or vending machines, successful interaction recognizes them as interactive partners. Clear, upfront communication not only saves time but consistently leads to superior results.

3. Discernment

Discernment represents the ability to thoughtfully evaluate what AI produces, how it produces it, and how it behaves within collaborative interactions. This critical skill encompasses three distinct dimensions: Product discernment focuses on evaluating the quality of actual outputs, assessing factors such as accuracy, appropriateness, coherence, and relevance. Process discernment involves examining how the AI arrived at its output, identifying potential logical errors, attention gaps, or inappropriate reasoning patterns. Performance discernment evaluates how the AI behaves within the collaboration process itself, considering whether its communication style, terminology usage, and responsiveness effectively serve the user's needs. Discernment operates in a continuous feedback loop with Description, creating an iterative process of communication and evaluation. Even the most advanced AI systems benefit significantly from human judgment and oversight, making discernment an essential competency for effective human-AI collaboration. Users can develop this skill by practicing evaluation across all three dimensions, providing specific feedback to AI systems, and leveraging their domain expertise to identify strengths and weaknesses in AI-generated content.

4. Diligence

Diligence encompasses taking responsibility for AI collaborations across three critical dimensions. Creation diligence involves being thoughtful about which AI systems to use and how to engage with them appropriately. Transparency diligence means being honest about AI's role in work with everyone who needs to know, ensuring appropriate disclosure based on context. Deployment diligence requires taking responsibility for verifying and vouching for the outputs that are used or shared with others. Different contexts—whether personal, academic, or professional—may have varying expectations for disclosure and verification standards. Thoughtful application of diligence helps ensure that AI collaborations are not only effective and efficient, but also ethical and safe, creating a foundation for responsible AI use that considers both immediate outcomes and broader implications for stakeholders and society.

Integrated Application

The 4Ds work synergistically to create comprehensive AI fluency. Effective Delegation sets the foundation for appropriate task distribution, while clear Description ensures productive communication. Discernment provides the critical evaluation skills necessary for quality assurance, and Diligence ensures that all AI collaborations remain ethical, transparent, and responsible. Together, these competencies enable users to harness AI's capabilities while maintaining human judgment, oversight, and accountability.

This course is developed by Rick Dakan and Joseph Feller, along with the Anthropic team. It's published on Anthropic's LMS Skilljar. You may access the course: AI Fluency: Framework & Foundations here. Additionally you may also read ValueCurve article on the need for frameworks.

Share ValueCurve

Subscribe now

The Blueprint for Pattern Recognition

Schema functions as an architectural blueprint that creates structured frameworks for data analysis. It provides a controlled framework necessary for consistent pattern recognition and replication. It's the difference between random observation and systematic discovery.

The Conceptual Foundation

Schema is the scientific method applied to data organization. It creates systematic classification frameworks that reveal underlying patterns in a data system.

Core Principles:

• Standardization: Ensure consistent measurement and categorization

• Validation: Provide quality control for pattern reliability

• Relationships: Define how different data elements connect and influence each other

• Evolution: Allow systematic growth and refinement over time

The Pattern Framework Components

To illustrate these principles in practice, consider a simple experiment tracking schema:

CREATE TABLE ExperimentResults (
    experiment_id INT PRIMARY KEY,           -- Unique pattern identifier
    hypothesis VARCHAR(500) NOT NULL,       -- Pattern prediction
    measurement_value DECIMAL(10,4),        -- Quantified observation
    confidence_level DECIMAL(3,2),          -- Pattern certainty (0.00-1.00)
    observation_date TIMESTAMP              -- Temporal pattern marker
);

Schema validation operates through systematic constraint rules that ensure pattern reliability and scientific rigor. Measurement values must fall within established scientific ranges, while confidence levels represent statistical significance thresholds. Temporal markers enable time-series pattern analysis, while researcher notes preserve observational context.

Beyond individual data integrity, schema defines relationship patterns between different elements, creating interconnected networks that mirror complex interdependencies found in natural systems. Well-designed schemas capture the web of connections that exist in real-world phenomena, enabling analysts to discover systematic relationships that drive meaningful insights.

Advanced Schema Architectures

Modern data analysis requires hierarchical pattern recognition that operates simultaneously across multiple levels of abstraction. This multi-layered approach enables organizations to understand both granular individual behaviors and macro-level market dynamics within a single analytical framework.

The power of this approach lies in its ability to identify cross-level relationships. Individual preference evolution patterns can aggregate to reveal market-wide trend shifts, while competitive positioning data can help explain individual customer lifecycle transitions.

How schema Enables AI/ML Development

Effective schema design forms the foundation upon which modern AI and machine learning systems build their pattern recognition capabilities:

1. Consistent Pattern Input: Algorithms require predictable data formats to identify reliable patterns at scale.

2. Validation Framework: Like peer review in scientific research, schema validation prevents unreliable data from corrupting pattern recognition processes.

3. Scalable Processing: Well-designed schemas allow AI systems to process millions of data points while maintaining pattern recognition accuracy.

4. Integration Capability: Schema standardization enables AI ecosystems where different systems can build upon each other's discoveries.

5. Evolution Tracking: Documents how patterns change over time, enabling AI systems to adapt and improve their recognition capabilities.

Traditional Data Analysis

Traditional data analysis represents the scientific method applied to pattern recognition—the systematic approach to extracting meaningful insights from structured observations. Rather than being obsolete in the age of AI, traditional analysis provides the foundational methods upon which modern AI/ML systems build.

The Scientific Pattern Recognition Process

Hypothesis Formation

Traditional data analysis starts with specific questions about patterns: "Do customers purchase more during certain seasons?" or "Can we predict equipment failure based on performance metrics?" This hypothesis-driven approach ensures analysis remains focused on meaningful patterns rather than random correlations.

Systematic Observation

Traditional analysis applies rigorous observation methods borrowed directly from experimental science: controlled data collection procedures, standardized measurement techniques, consistent categorization systems, and temporal pattern tracking.

Pattern Quantification

Mathematical methods quantify observed patterns with precision through descriptive statistics, correlation analysis, regression modelling, and time series analysis.

Pattern Recognition Methods

Descriptive Analysis

Descriptive analysis represents the archaeological phase of data exploration—understanding what patterns exist in current data before attempting to predict future outcomes. These descriptive patterns become the foundation for predictive modeling, revealing the data formations most likely to contain valuable predictive patterns.

Comparative Analysis

Comparative analysis identifies differences between groups or time periods to understand pattern variations and their underlying causes. This analysis reveals not just what patterns exist, but why they exist—enabling more sophisticated AI models that can account for causal relationships.

Predictive Analysis

Predictive analysis uses historical patterns to forecast future occurrences with quantified confidence levels, forming the bridge between traditional analysis and automated AI systems.

For example, a customer churn model might identify that usage decline greater than 40% indicates 75% churn probability within 90 days, while combined factor models can achieve 92% prediction accuracy.

These predictive models form the conceptual foundation that AI/ML algorithms later automate and scale. The traditional analysis establishes the logical framework, while AI systems provide the computational power to apply these frameworks across vast datasets in real-time.

Share ValueCurve

The key aspect of collecting data is identifying patterns, hence similar to math or rules of grammar. Patterns, once known can be replicated for similar future results. Hence pattern recognition from existing data helps in predicting future occurrences.

What is Data?

To harness this digital oil effectively, we must first understand its fundamental nature. Data refers to facts, figures, or information collected for analysis. It represents observations or measurements that can be processed to extract insights, make decisions, or build models. Data can exist in various forms like numbers, text, images, or sounds.

Understanding data as a scientific tool is the skill that's fundamental to learning AI/ML development models, thereby its systems. Just as oil exists in different geological formations requiring different extraction methods, data comes in various forms and shapes, each demanding unique analytical approaches.

Examples of data as pattern repositories:

• Stock market fluctuations revealing economic cycles

• Weather measurements showing seasonal patterns

• Customer behavior data indicating purchasing trends

• Medical records exposing disease progression patterns

• Social media interactions demonstrating communication patterns

Three Data Types

Data manifests in three fundamental architectures, each requiring different extraction and refinement techniques.

1. Structured Data

Structured data represents the most refined form of information organization, following strict organizational principles with precise, predictable rules.

Predictable format enables algorithmic pattern detection, allowing systems to automatically identify recurring structures and anomalies within the data. Consistent schema allows for mathematical operations to be performed reliably across datasets, ensuring that calculations and transformations maintain their integrity. Tabular organization facilitates statistical analysis by providing a structured framework where variables can be easily compared and measured. Additionally, relationships between data points are explicitly defined, making it possible to trace connections and dependencies throughout the entire dataset.

Real-world Example:

Consider how structured data reveals patterns in customer behavior through organized transaction records:

Customer Transaction Patterns:
Customer_ID | Purchase_Date | Amount  | Product_Category | Season
    001     | 2024-01-15   | $150.00 | Electronics     | Winter
    001     | 2024-03-20   | $85.50  | Clothing        | Spring
    001     | 2024-07-10   | $220.00 | Sports          | Summer

This structure reveals purchasing seasonality patterns - a machine learning algorithm can identify that Customer 001 increases spending in summer months, enabling predictive models for inventory and marketing.

2. Unstructured Data

Unstructured data represents information in its most natural, human-generated form. Like studying animal behavior in the wild rather than in controlled laboratory conditions, unstructured data captures the complexity and richness of real-world phenomena. This natural complexity presents unique analytical challenges that mirror the unpredictable nature of organic information flow.

Patterns exist in unstructured data but are embedded within intricate contexts, making them difficult to isolate and analyze without considering the surrounding environmental factors. Multiple pattern types can coexist in single data points, creating layered complexity where different analytical approaches may be needed to uncover each distinct pattern.

This multifaceted nature requires sophisticated algorithms to extract meaningful information, as traditional simple pattern matching techniques often prove inadequate for such elaborate data structures. Furthermore, the data contains both explicit and implicit information patterns, where some relationships are clearly visible while others remain hidden beneath the surface, requiring deeper analytical techniques to reveal their significance.

Mining unstructured data is like an exploration - valuable insights are buried beneath layers of complexity. Each document, image, or conversation contains multiple pattern layers that sophisticated AI systems can now excavate and analyze.

Examples of Hidden Patterns:

The complexity of unstructured data becomes evident when analyzing customer feedback:

Customer Review Analysis:
"This smartphone camera is incredible! Battery life could be better though. 
After 6 months of heavy usage, still works great. Would recommend for photography 
enthusiasts but maybe not for heavy gamers due to heating issues."

Extractable Patterns:

• Sentiment: Mixed (positive camera, negative battery)

• Product lifecycle: 6-month durability confirmed

• Product issues: Battery life and heating concerns

• Recommendation pattern: Conditional endorsement

  Share

3. Semi-Structured Data

Semi-structured data represents a bridge between organized systems and natural complexity. Semi-structured data combines organizational elements with content variability such as an email or a log file entry below :

Server logs often follow patterns like: 
[2024-08-24 14:30:15] ERROR user_id:12345 action:login_attempt ip:192.168.1.100 message:"Invalid password - 3rd attempt" 

structured timestamp and fields, but flexible message content.

This hybrid approach offers significant pattern recognition advantages. Structural tags provide navigation markers that help both systems and users locate specific information within the data, creating a roadmap through otherwise complex content. Content remains flexible for natural expression, allowing information to maintain its original context and meaning while still being organized in a systematic way.

This approach achieves machine-readable organization with human-readable content, striking an optimal balance where automated systems can process the data efficiently while humans can still interpret and understand it intuitively. This dual accessibility enables both automated and manual pattern analysis, giving researchers and analysts the flexibility to choose the most appropriate analytical approach based on their specific needs and objectives.

Key Insights

• Data is pattern-rich information waiting to be discovered through systematic exploration

• Different data types require different extraction techniques but serve the same pattern recognition purpose

• Structured data provides immediate accessibility to pattern recognition algorithms

• Unstructured data contains deeper insights but requires more sophisticated exploration methods

• Semi-structured data offers balanced accessibility with rich contextual information.

Stay tuned for Part 2, where we'll uncover how proper organization and systematic analysis turn these data forms into the foundation of artificial intelligence.

Subscribe now

Subscribe now

Here's a detailed explanation based on the general understanding and updates associated with CSF 2.0:

Core Functions of NIST CSF 2.0:

• Identify: This function involves understanding and managing cybersecurity risks in the context of the organization's mission, assets, data, and capabilities. It includes developing an organizational understanding, risk management strategy, and conducting a risk assessment.

• Protect: Focuses on developing and implementing appropriate safeguards to ensure delivery of critical infrastructure services. This includes identity management, access control, awareness and training, data security, information protection processes, and maintenance.

• Detect: Enhancements in CSF 2.0 might include more detailed practices for continuous security monitoring, which involves identifying the occurrence of a cybersecurity event. This function ensures timely detection of anomalies or incidents.

• Respond: This function outlines the activities an organization engages in when a cybersecurity incident is detected. CSF 2.0 could emphasize more on incident response planning, communication, analysis, and mitigation activities.

• Recover: It aims at restoring any capabilities or services that were impaired due to a cybersecurity incident. This involves recovery planning, improvements, and communications.

New Elements or Enhancements in CSF 2.0:

• Govern: While not explicitly mentioned in previous versions as a core function, discussions around CSF 2.0 have highlighted the importance of governance. This could include cybersecurity policies, processes, and procedures, which guide the organization's approach to managing cybersecurity risks.

• Implementation Guidance: There's a noted enhancement in providing more detailed implementation examples, which separates from the core framework, allowing for regular updates without revising the entire document.

• Profiles: Organizations create "Current" and "Target" Profiles to assess their cybersecurity maturity and alignment with business requirements. CSF 2.0 offers more nuanced guidance on creating these Profiles, emphasizing how they can be used to drive cybersecurity improvements.

• Tiers: These help in understanding the organization's cybersecurity risk management practices' sophistication. CSF 2.0 refines how these Tiers are evaluated, possibly integrating more real-world scenarios or industry-specific nuances.

• Risk-Based Approach with Emphasis on Outcomes: While maintaining its core risk-based approach, CSF 2.0 focuses more on outcomes rather than prescriptive methods, allowing organizations to innovate within their cybersecurity strategies.

Community and Organizational Profiles:

• CSF 2.0 introduces or elaborates on the concept of Organizational and Community Profiles. Organizational Profiles help in aligning cybersecurity activities with organizational goals, while Community Profiles assist in sharing cybersecurity practices across similar sectors or communities, fostering a collective approach to cybersecurity.

Key Considerations:

• Scalability: One of the strengths of CSF 2.0 is its scalability, applicable from small businesses to large enterprises across different sectors, which are reflected in tailored guidance or examples within the framework.

• Flexibility: The framework continues to be voluntary, encouraging adaptation rather than strict compliance, which is crucial for its global acceptance and implementation.

• Continuous Improvement: There's an implicit push in CSF 2.0 towards viewing cybersecurity as a continuous process, not a one-time setup, which aligns with the evolving nature of cyber threats.

The NIST CSF 2.0, through these enhancements, aims to provide a more adaptable, scalable, and actionable framework for managing cybersecurity risks, reflecting the dynamic nature of cyber threats and technological advancements. This version likely encourages organizations not only to secure their environments but also to engage in governance that promotes a culture of cybersecurity awareness and proactive risk management.

For more details visit : NIST

Share ValueCurve

Subscribe now

Banks have to find the optimal balance point based on their risk tolerance, customer base, and competitive position. The other case in point is the display of recommendation engines that drive user engagement & display ad revenue. Engagement-optimized models tend to promote polarizing, sensationalist, or addictive content, even if it's borderline misinformation or emotionally manipulative.

Smart Systems

Nevertheless, it's expected of any production ML system to display following characteristics :

Scalability : ML Models can grow complex fairly quickly. It can be because the number of parameters used to make predictions has increased from 100 million to a Billion. The data volumes have increased due to increased user traffic or there's the addition of more features. ML Systems should be able to Scale up as well as scale down in terms of resource requirements such as CPU, GPU, Memory and network bandwidth, they should also be able to handle growth in the number of ML models or algorithms designed to meet various business objectives that scale out .

Maintainability : As the system & the underlying infrastructure grows and increases in complexity, these should be maintained with the help of documentation, code review processes, data cleansing and versioning of artefacts. As the standards, teams and operations grow, regularly carrying out maintenance activities, training of team members and refactoring of the code should be conducted.

Adaptability : Systems should have the capacity to discover performance management and updates without service interruption. Cloud-based platforms like AWS SageMaker, Google Cloud AI, and Azure ML implement blue-green deployments and rolling updates that maintain service continuity during model transitions. These systems actively monitor model performance metrics and can detect degradation through automated alerts, though they typically require human intervention for complex decision-making. Production systems at companies like Netflix and Uber showcase gradual traffic shifting between model versions, allowing new models to be tested on small user segments before full rollout.

Reliability : System reliability is paramount in production ML environments, requiring robust design that maintains functionality despite various failure modes.

The system must gracefully handle hardware failures through redundancy and failover mechanisms, ensuring continuous operation when servers crash or network connections drop. Software faults, including bugs in code updates or dependency conflicts, should be contained through proper error handling and rollback capabilities. Human errors, such as incorrect configuration changes or accidental data deletions, need safeguards like automated backups, configuration validation, and staged deployment processes.

By implementing comprehensive fault tolerance, monitoring systems, and recovery procedures, ML systems can maintain their desired performance levels and continue delivering accurate predictions even when facing unexpected adversities, ensuring business continuity and user trust.

Iterative Process

Deploying an ML system marks the beginning, not the end, of its lifecycle. Unlike traditional software that remains relatively stable post-deployment, ML systems require continuous iteration due to their dynamic nature. Real-world data constantly evolves, causing model performance to degrade over time through data drift and concept shifts. Production deployment reveals new edge cases, user behaviours, and system bottlenecks that weren't apparent during development.

Teams must establish robust monitoring frameworks to track model accuracy, latency, and business metrics in real-time. Regular retraining with fresh data, feature engineering improvements, and architecture optimizations become essential maintenance tasks. This iterative approach ensures the system adapts to changing conditions, maintains reliability, and continues delivering business value throughout its operational lifetime.

For additional reading, refer : Designing Machine Learning Systems by Chip Huyen .

Share ValueCurve

Subscribe now

I realize, I’ve already mentioned some of the terms that sound geek ( or Greek & Latin if you prefer that ). In this article I discuss the basics of Cryptography & my source is the book The Basics of Bitcoin & Blockchains by Antony Lewis.

Let’s begin by understanding what’s Cryptography ?

Cryptography is all about sending secret messages that can be read only by the intended recipient. Now, when the messages are shared over the internet its’ not very difficult for eavesdroppers to snoop on the messages. More so if the information has financial implications, such as, if Alif is transferring electronic money to Behrouz - he wouldn’t want Sasha, Saha or Sahira to know the amount of money that has been transferred or may be even the information that he has transferred anything at all for that matter.

How does one achieve this over the internet or any network of computers that are interconnected?

To achieves this, a message is encrypted mathematically using software programs by the originator . Once these messages are received by the recipient these are decrypted .

In a nutshell :

Encryption is the process of turning a plaintext (i.e., readable ) human message into cyphertext, so that if the encrypted message is intercepted a snooper can’t understand it.

Decryption is the process of turning the cyphertext back into readable plaintext.

This process of encryption & decryption is Cryptography .

There are 3 main components in Cryptography:

1. Key Pair

2. Hashing

3. Digital Signature

(image source : SpringerOpen)

Let’s look at each of these components in detail :

Key Pair

A key pair consists of two mathematically linked keys : a public key & a private key . Public key can be shared with the world & anyone can use it to encrypt messages for you. The private key is known only to you to decrypt those messages .

PGP ( Pretty Good Privacy ) is a scheme developed in the 1990’s for encrypting, decrypting and digitally signing email messages.

Bitcoin uses a different scheme called ECDSA - Elliptic Curve Digital Signature Algorithm .

Hashing

A hash function is a series of mathematical steps or algorithms that you an perform on some input data, resulting in a fingerprint, or digest, or simply, a hash .

A cryptographic hash function is special & has some characteristics that makes it useful in cryptography & for cryptocurrencies - such as ;

• It is deterministic - so the same message always results in the same hash .

• It’s quick to compute.

• It’s not feasible to generate a message from its hash .

• A small change to a message should change the hash value extensively so that the new hash value appears uncorrelated with the old hash value .

Digital Signatures

A digital signature is created by taking the message you want to sign and applying a mathematical formula with your private key . Anyone who knows your public key can mathematically verify the that this signature was indeed created by the holder of the associated private key ( but without the knowing the private key ) .

So, anyone can independently validate that this piece of data was signed by the private key holder of this public key .

Message ( Digest Function/ Hash ) + Private Key → Digital Signature

Message + Digital Signature + Public Key → Valid/ Invalid

Digital Signatures are a subset of e-signatures & are used in blockchain transactions because they prove account ownership, and the validity of a digital signature can be proven mathematically and offline, without asking any other party.

There is more to discuss on the topic of cryptocurrencies, that we shall in near future. If you have more book or research articles references to share, please post them in comments.

P.S : Antony Lewis recommends reading Steven Levy’s book Crypto to learn the in-depth story of PGP.

Share ValueCurve

In the evolving landscape of developer tooling, CodeRabbit.ai automates the manual, tedious process—code review. Founded in 2023 and headquartered in Walnut Creek, California, the company has gained traction among open-source communities and enterprise teams looking to streamline their Git workflows.

CodeRabbit is an agentic code review platform designed to understand context, syntax, and intent across pull requests. With features like line-by-line review, semantic summaries, bug detection, and one-click fixes, CodeRabbit integrates seamlessly into platforms such as GitHub, GitLab, Azure DevOps, and Bitbucket.

It has earned adoption across projects like NextUI, Novu, Appsmith, and Pipedream, while boasting over 600 paying organizations and 300K+ PRs reviewed. The platform recently raised $16M Series A round (led by CRV and Flex Capital) pushing its total funding to $19.6 million, that marks a significant milestone for a company barely two years old.

Unlike code editors such as Cursor AI, which focus on the developmental side of coding through autocomplete and natural language prompts, CodeRabbit is focused on quality assurance, performance, and editorial coherence. While Cursor AI enhances coding experience as an IDE built on VS Code, CodeRabbit acts like a reviewer—offering security insights via Semgrep, sprint summaries, and changelog generation that tie directly into branching architecture.

Workflow

At the heart of CodeRabbit’s workflow lies the pull request (PR)—a formal mechanism for proposing changes to codebases hosted on Git platforms. PRs allow developers to merge branches after peer review, often triggering automated CI/CD tests, security scans, and performance evaluations. CodeRabbit enhances this process by integrating AI-driven feedback that mimics senior developer insight—making suggestions, highlighting inconsistencies, and even learning team preferences over time.

The process:

1. Create a feature branch.

2. Commit changes.

3. Push to a remote repository.

4. Open a pull request.

5. Discuss, review, iterate.

6. Merge once approved.

By streamlining this cycle, CodeRabbit helps developers catch bugs earlier, adhere to style guidelines, and reduce the review burden that slows down deployment velocity.

Smart Fit

As development teams grow, staying on top of code quality, consistency, and collaboration becomes a real challenge. That’s where CodeRabbit steps in—automating the repetitive parts of code review adding human-like insight across Git workflows.

CodeRabbit makes it easier to:

• Speed up code reviews without sacrificing quality

• Catch bugs early with AI-powered scanning

• Generate release notes and team summaries directly from pull requests

• Enforce consistent standards across different branches and collaborators

• Learn team preferences over time, so feedback gets more relevant

It's especially helpful while using platforms like GitHub, GitLab, or Azure DevOps, and developers want a tool that works with their existing branching strategies and CI/CD pipelines. CodeRabbit helps remove the review bottleneck and ship the code faster.

If you’ve worked with CodeRabbit—or any similar platform focused on code review automation—feel free to share your experience in the comments. Whether you’ve found particular features useful or encountered limitations, your feedback can help others explore how these tools fit into different team structures and development workflows.

Subscribe now