Machine Learning (ML) · Beginner level

Beginner level · Module 3: First Models: Regression and Classification · Lesson 2 of 3

Classification: Logistic Regression and k-Nearest Neighbours

Predict categories with logistic regression and k-nearest neighbours, and see why feature scaling matters.

In this lesson you will

  • Train logistic regression and k-NN classifiers
  • Explain predicted probabilities and the need for scaling

Key concepts

logistic regressionpredicted probabilityk-nearest neighboursfeature scaling

The lesson

Logistic regression estimates the probability of a class, such as 0.73 chance of dropout, and turns it into a decision with a threshold, usually 0.5. The threshold can be changed to suit the cost of each kind of mistake.

k-nearest neighbours (k-NN) classifies a new case by looking at the k most similar training cases and taking a vote. It is simple to understand and needs no training step beyond storing the data.

k-NN measures distance, so features on large scales dominate: rainfall in millimetres swamps temperature in degrees. Scale features to similar ranges with StandardScaler before using distance-based models.

Step by step

  1. Train logistic regression and print predicted probabilities.
  2. Change the threshold and observe the predictions.
  3. Train k-NN with and without scaling.
  4. Compare their accuracy with the majority-class baseline.
Code example

k-nearest neighbours from scratch: which crop for this plot?

A ten-line k-NN that recommends a crop by finding the three most similar demonstration plots.

Pythonknn_crops.py
from collections import Counterfrom math import dist # (rainfall in mm, mean temperature in °C) -> crop that did best on demonstration plotstraining = [    ((800, 27), "sorghum"), ((850, 28), "sorghum"), ((700, 29), "sorghum"),    ((1200, 25), "maize"), ((1300, 24), "maize"), ((1150, 26), "maize"),    ((1600, 23), "cassava"), ((1700, 24), "cassava"),] def predict(point, k=3):    nearest = sorted(training, key=lambda item: dist(point, item[0]))[:k]    votes = Counter(label for _, label in nearest)    return votes.most_common(1)[0][0], [label for _, label in nearest] for plot in [(900, 28), (1250, 25), (1550, 24)]:    crop, neighbours = predict(plot)    print(plot, "->", crop, neighbours)

Output

(900, 28) -> sorghum ['sorghum', 'sorghum', 'sorghum']
(1250, 25) -> maize ['maize', 'maize', 'maize']
(1550, 24) -> cassava ['cassava', 'cassava', 'maize']

How it works

math.dist gives the straight-line distance between two (rainfall, temperature) points. Sorting the training plots by that distance and keeping three gives the nearest neighbours, and Counter takes the majority vote. The third plot's neighbourhood is mixed — two cassava plots and one maize plot — so the vote is 2 to 1. Notice that rainfall (hundreds of mm) dominates temperature (a few degrees) in the distance: real k-NN models scale features first, as the lesson explains. Data is invented; the output was verified with an equivalent JavaScript implementation because no Python interpreter was available when the example was written.

Key concepts

distance between feature vectorsk nearest neighboursmajority votewhy feature scaling matters

Practice activity

Train both classifiers on the dropout data and explain which one you prefer and why.

Expected result: Two trained classifiers compared against a baseline, with scaling applied correctly for k-NN.

Check your understanding

  1. What does logistic regression output before the threshold is applied?

    Show answer

    A probability for the class.

  2. Why scale features for k-NN?

    Show answer

    Distances are dominated by features with large numeric ranges.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.