The Complete Guide to Securing SSH on Your Server

If you run a Linux server, you probably use SSH to connect. Attackers know that too. The second port 22 is open to the internet, bots start hammering it with brute-force login attempts. I’ve watched fresh servers rack up thousands of attempts within hours of going online.

Weak passwords, outdated cryptography, or a sloppy config can turn that background noise into a real problem. Data breach. Ransomware. Your box getting drafted into someone else’s botnet. All of that is on the table when SSH is left soft.

So securing SSH is the first thing you do on any new server, not the last. A handful of config changes raises the bar high enough that most attackers move on to easier targets.

By the end of this guide, you’ll have a checklist you can run through on every server you touch. Don’t wait until you’re done reading. Apply each step as you go.

Protectli FW6C/FW6D This fanless firewall appliance is built for pfSense and OPNsense, which gives you a clean place to gate SSH access for a homelab or media server. Intel NICs and hardware AES-NI keep routing fast under load, and the firewall rules are flexible enough to lock SSH down to specific source IPs or a VPN subnet. A solid pick if you want a dedicated edge device handling remote access instead of trusting your ISP router.

Contains affiliate links. I may earn a commission at no cost to you.

Amazon

A firewall appliance like that solves the network side of the problem. The rest of the work happens on the server itself, starting with how you authenticate.

Step 1: Generate a Secure SSH Key Pair

Password authentication over SSH is like leaving your house key under the doormat. It works, sure. Every bot on the internet is also happy to spend all day guessing your password. Key-based authentication is practically unbreakable when you use modern algorithms.

To set up key-based authentication, you need to create the keys first. This works the same on Linux, macOS, and Windows (PowerShell or WSL).

Run this on your client machine to create the key pair:

ssh-keygen -t ed25519

The ed25519 algorithm generates keys that are both smaller and more secure than the older RSA standard. If you’re stuck on an older system that doesn’t support ed25519, fall back to this:

ssh-keygen -t rsa -b 4096

This creates two files: a private key (which you guard with your life) and a public key (which you can share freely). By default, both keys land in your .ssh folder (~/.ssh/ on Linux and macOS, C:\Users\YourName\.ssh on Windows).

Step 2: Deploy Your Public Key to the Server

Now you need to tell the server to trust your public key.

Automatic method (the easy way):

ssh-copy-id user@server_address

This does all the heavy lifting for you. It copies your public key and appends it to the right place on the server.

Manual method (when you want control):

1. Copy the contents of your ~/.ssh/id_ed25519.pub file
2. Append it to the server’s ~/.ssh/authorized_keys file

Here’s the thing. Your first login still needs your password because the server hasn’t seen your key yet. Once the key is in place, you can disable password authentication entirely and never type that password again for SSH.

MINISFORUM MS-A2 Dual 10GbE SFP+ and dual 2.5GbE on a compact box makes the MS-A2 a strong fit for a homelab gateway or jump host. The networking headroom means you can run a hardened SSH server, a VPN endpoint, and a handful of containers without choking the link. Flexible storage and modern Ryzen silicon also give you room to spin up extra services as your lab grows.

Contains affiliate links. I may earn a commission at no cost to you.

Amazon

Once your hardware is sorted and the key is on the server, the next job is to slam the SSH daemon’s defaults shut.

Step 3: Secure the SSH Configuration

The real power of SSH security lives in /etc/ssh/sshd_config. This is where you harden the entry point and raise the bar high enough that attackers move on to softer targets.

Open it with:

sudo nano /etc/ssh/sshd_config

Here are the most important changes:

Disable password authentication
This kills brute-force attacks dead.

PasswordAuthentication no

Disable root login
Never let root log in directly through SSH. Log in as your regular user and escalate with sudo when needed.

PermitRootLogin no

Change the SSH port (optional)
Moving off port 22 cuts down on automated bot noise. It won’t stop a determined attacker, but it cleans up your logs.

Port 2222

If you change this, update your firewall rules at the same time. Don’t leave it for later.

Enforce modern encryption
Force SSH to use strong ciphers and message authentication codes. This prevents fallback to weaker algorithms.

Ciphers aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-256,hmac-sha2-512

Restrict user access
Limit SSH to specific users or groups instead of allowing every account on the box:

AllowUsers youruser

After making these changes, restart SSH to apply them:

sudo systemctl restart ssh

On some distributions the service is named sshd instead of ssh. If the first command errors out, try sudo systemctl restart sshd.

Your SSH config is now meaningfully tighter than the defaults. These settings layer together, so an attacker who beats one still has to beat the others.

Step 4: Set Correct File Permissions

SSH has strong opinions about file permissions, and it’s not shy about telling you when they’re wrong. If your .ssh directory or its files are too open, SSH will flat-out refuse to use them. Think of it as SSH protecting you from yourself.

Here are the correct permissions:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519
chmod 644 ~/.ssh/id_ed25519.pub
chmod 600 ~/.ssh/authorized_keys

Breaking this down:

• 700 on the .ssh directory means only you can read, write, or enter it
• 600 on private keys means only you can read or write them
• 644 on the public key allows others to read it, which is fine because it’s public
• 600 on authorized_keys keeps it private to your account

If you’re seeing “bad owner or permissions on .ssh/config” errors, these settings will almost certainly fix it. SSH is picky about security, but once the permissions are right it works reliably.

RaspberryPi 4GB A Pi 4 with 4GB is a cheap, low-power way to practice SSH hardening on real hardware without putting a production server at risk. Spin up Ubuntu Server or Raspberry Pi OS, run through every step in this guide, and break it on purpose so you know what the failure modes look like. Once you’re confident, repeat the playbook on the box that actually matters.

Contains affiliate links. I may earn a commission at no cost to you.

Amazon Newegg

With the host hardened, you still need to think about what can reach it from the network.

Step 5: Harden Network Access

Your SSH service is now using keys and saner settings. Good. But if attackers can reach it from anywhere on the internet, you need more layers. Great lock on the front door, sure. Now stop putting the door on the street.

Firewalls
Limit SSH to specific source IPs where you can. This is your first line of defense. Example using UFW (Uncomplicated Firewall):

sudo ufw allow from 203.0.113.10 to any port 2222

VPN or Bastion Host
Put SSH behind a VPN. WireGuard or OpenVPN both work. This stops the whole internet from probing every machine you own. Only hosts on the VPN can see SSH at all.

Fail2ban
Fail2ban scans logs and bans IPs that show malicious behavior, like repeated failed login attempts. Install it on Debian or Ubuntu with:

sudo apt install fail2ban

Then configure /etc/fail2ban/jail.local for the sshd jail. Even with passwords disabled, Fail2ban still blocks port-scanning noise and keeps your logs cleaner. Think of it as a bouncer who remembers troublemakers and won’t let them back in.

[DEFAULT]
# Whitelist: your trusted networks and admin IPs that should never be banned.
# Replace the placeholders with your real IPs/subnets.
ignoreip = 127.0.0.1/8 ::1 192.168.1.0/24 10.0.0.0/24

# How long to ban an offender (and enable incremental bans).
bantime = 1h
bantime.increment = true
bantime.factor = 2
bantime.maxtime = 1w

# How far back to count failures, and how many failures trigger a ban.
findtime = 10m
maxretry = 5

# Use the systemd journal for log parsing.
backend = systemd

# Use nftables if your distro uses it. Fall back to iptables if needed.
banaction = nftables-multiport

# -----------------------------------
# SSH JAIL
# -----------------------------------
[sshd]
enabled = true
filter  = sshd

# If you changed the SSH port, reflect it here (e.g., port = 2222).
port    = ssh

# Mode “aggressive” catches more patterns (invalid users, many auth noise cases).
# Requires fail2ban 0.11+ with newer sshd filter.
mode    = aggressive

Step 6: Test and Verify

Now comes the moment of truth. Before you close that original SSH session (your safety net), make sure everything actually works:

1. Open a new terminal window and test logging in with your key.
2. Confirm you can’t log in as root.
3. Confirm password login is refused.
4. Check logs with:

   sudo tail -f /var/log/auth.log
   

   On systemd distributions without auth.log, use sudo journalctl -u ssh -f (or -u sshd).

This is where patience pays off. Only after confirming everything works should you close your original session. If something breaks, you still have that old session open to fix it. There’s nothing quite like the sinking feeling of being locked out of your own server because you skipped this step.

Step 7: Ongoing Monitoring and Auditing

SSH hardening isn’t a one-and-done deal. Think of it like home security. You don’t install locks and then never check them again. Attackers adapt their methods, so your defenses need regular attention too.

Here’s what to keep an eye on:

• Review logs regularly (/var/log/auth.log or journalctl -u ssh). Look for failed login attempts, especially repeated ones from the same IP addresses.
• Audit authorized keys at least monthly. Remove old keys, particularly when team members leave or change roles. Stale keys are like forgotten spare house keys under the doormat.
• Check for anomalies like new user accounts you didn’t create or SSH configuration changes you didn’t make. These can signal that someone is already in.
• System updates: keep the box patched. OpenSSH gets security updates often, and missing one defeats everything else you’ve done. On Debian or Ubuntu, run sudo apt update && sudo apt upgrade on a schedule, or use unattended-upgrades for the security set.

Set a calendar reminder to do this monthly. It takes maybe 10 minutes. Catching problems early beats dealing with a breach later.

Troubleshooting Common SSH Problems

Even experienced admins lock themselves out occasionally. Here are the most common ways things go sideways and how to fix them:

• Locked out after disabling passwords
  Classic mistake. Your SSH key wasn’t copied correctly before you disabled password authentication. Get into the server through your hosting provider’s web console or rescue environment, then re-enable PasswordAuthentication yes in sshd_config until you have your keys working properly.

• Wrong file permissions
  If you see errors like “bad owner or permissions on .ssh/config”, your SSH files are readable by other users, which SSH treats as a security risk. Reset the permissions as shown in Step 4.

• Forgot to update firewall when changing port
  Changed your SSH port to Port 2222 but forgot to open that port in the firewall? Your connection will hang, looking like a network issue. Always update firewall rules before changing the SSH port, not after.

• Trying to allow root login with keys
  Even with “key-only” authentication, allowing direct root login creates unnecessary risk. If someone compromises your key, they have immediate root access. Stick with PermitRootLogin no and use sudo instead.

• VPN misconfiguration
  If you’re routing SSH through a VPN, test access thoroughly in a non-production environment first. The added complexity can leave you stranded if you misconfigure it.

ASROCK Mini-Desktop Computer This compact barebone system gives you a tidy place to run lightweight VMs or containers and isolate SSH services on their own host. Modern Intel CPUs and flexible storage make it a practical platform for a dedicated SSH gateway or jump box that sits between your homelab and the outside world. Quiet, efficient, and small enough that you’ll actually keep it powered on.

Contains affiliate links. I may earn a commission at no cost to you.

Amazon Newegg

A dedicated jump host is a nice luxury, but it doesn’t change the questions readers ask most often. Here are the ones that keep coming up.

FAQs: Secure SSH in Practice

Conclusion

If you take one thing from this guide, it’s this. SSH security depends less on any one trick and more on layers. Keys instead of passwords, no root logins, careful configuration, restrictive firewalls, monitoring, and patching all stack together to drastically lower your risk.

Set aside time to secure SSH right after deploying any new server. It pays off later, whether you’re running a personal project or production infrastructure. Dealing with a compromised box is far worse than spending 30 minutes hardening SSH upfront.

Next steps worth exploring:

• Automating SSH hardening with configuration management tools like Ansible.
• Adding multi-factor authentication for SSH (Google Authenticator PAM or a hardware key like YubiKey).

Run through this checklist on every new server. After a few rounds, the steps stop feeling like a chore and start feeling like muscle memory.

Other Sources

1. OpenSSH man pages and docs
2. Ubuntu Community Security Guide on SSH
3. Red Hat Security Hardening for SSH
4. Fail2ban Documentation
5. DigitalOcean SSH Key Tutorial
6. Server Fault: SSH File Permission Pitfalls