Introduction
A Proxy Server is an intermediary or a middle man server that retrieves data from Internet sources on behalf of the user. They act as a protective barrier by adding an extra layer of security to shield users from potential internet threats.
Proxy servers have diverse applications, which vary based on their configuration and type. They are commonly used for anonymous web browsing, bypassing geographical restrictions, and managing web traffic.
It is important to note that like any other internet-connected device, proxies come with cybersecurity risks. Users should be mindful of these risks before utilizing proxy services.
How does it work?
In a typical scenario, a user accesses a website by directly requesting the web server, from a web browser via their IP address. The web server then sends a response containing the website data directly back to the user.
A proxy server conceals the IP address of the client from the server by using a different IP address on behalf of the user. Thus acting as an intermediary between the user and the web server. When sending a request from a proxy server the process is different.
The user sends a request to access a website through their browser.
The request is received by the proxy server.
The proxy server forwards the request to the web server on behalf of the user.
The web server responds by sending the requested data to the proxy server.
Finally, the proxy server forwards the response from the web server to the user.
Proxy Server vs VPN's
Proxy servers and Virtual Private Networks (VPNs) are often confused, but they serve different purposes. While both can help with privacy and accessing restricted content, VPNs offer encryption for all internet traffic, providing a higher level of security. In contrast, a proxy server does not encrypt data. Since all traffic is routed through a VPN server, it tends to be slower compared to a proxy server. Additionally , some proxy servers can also cache data, improving their speed by storing frequently accessed information.
Enough with the chatting, let's dive into the code!๐
Building Your Own Proxy Server
In this section, we'll explore how to create a simple proxy server using the Go programming language. This section assumes that you have some prior experience in Golang or at least have a basic knowledge of Golang. By building our own proxy server, we'll gain a better understanding of how they work and how to customize them to suit our needs. Let's get started!
This tutorial utilizes Gin, a lightweight web framework for Go, to develop a simple server. This server will serve endpoints required for our proxy server. Gin is well-suited for this task due to its simplicity and efficiency in handling HTTP requests and responses.
Create a main.go file in your project repository, you can use a text editor or an Integrated Development Environment (IDE) like Visual Studio Code, Sublime Text, or Atom. Follow these steps:
Open your text editor or IDE.
Create a new file and save it as
main.goin your project directory.
Here's the main.go file for your proxy server:
package main
import (
"log"
"github.com/Uttkarsh-Raj/Proxie/routes"
"github.com/gin-gonic/gin"
)
func main() {
router := gin.New() // Initialize a new router
router.Use(gin.Logger()) // Helps you use the inbuilt logger from the gin framework
routes.IncomingRoutes(router) // Initialize the routes (later in this tutorial)
log.Fatal(router.Run(":7000")) // Start the server and if error occurs log the issue
}
To start the server, open your terminal and run the following command:
$ go run main.go
This command will compile and execute your Go program, starting the server. You should see output indicating that the server is running. The only problem is that we have not yet defined the IncomingRoutes() function.
Now we need to add routes to your server using the Gin framework. Routes in a web server define how the server responds to incoming requests. Each route is associated with a specific HTTP method (e.g., GET, POST, PUT, DELETE) and URL path, and is handled by a corresponding handler function. Create a new directory named routes in your project directory and add a routes.go file inside it.
Add the following code to routes.go:
package routes
import (
"github.com/Uttkarsh-Raj/Proxie/controller"
"github.com/gin-gonic/gin"
)
func IncomingRoutes(router *gin.Engine) {
router.GET("/", controller.ProxyServer()) // The handler function is in the next section
}
From the code snippet above, it's evident that the IncomingRoutes function takes a pointer to the router and adds a GET method to it. This method will be triggered when a request is made to the default route /.
After defining the routes, we need to specify the actions to be taken when these routes are called. To do this, we'll create a controller directory and add a controller.go file to it. We'll then paste the following code into this file.
package controller
import (
"context"
"fmt"
"io"
"net"
"net/http"
"net/url"
"strings"
"time"
"github.com/gin-gonic/gin"
)
func ProxyServer() gin.HandlerFunc {
return func(c *gin.Context) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*60)
defer cancel()
// Get the query URL
queryURL := c.Query("url")
if queryURL == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Missing query parameter 'url'"})
return
}
// Convert to a uri
targetUrl, err := url.Parse(queryURL)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error parsing target URL: %s", err)})
return
}
// Create a new client to send the request using this servers context
// Request the target url
client := &http.Client{}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, targetUrl.String(), c.Request.Body)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error creating request: %s", err)})
return
}
resp, err := client.Do(req)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error connecting to the destination server: %s", err)})
return
}
defer resp.Body.Close()
// Copy the response and the headers received
c.Status(resp.StatusCode)
for k, v := range resp.Header {
c.Header(k, v[0])
}
// Copy the response body
if _, err := io.Copy(c.Writer, resp.Body); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error connecting to the destination server: %s", err)})
return
}
}
}
This will complete the implementation for the basic proxy server. To run this server, run the command in the cli :
$ go run main.go
Now that your proxy server is up and running, you can test it by configuring your browser to use the proxy server's port. To demonstrate, I'll show you an example where I'll fetch my IP address first without using the proxy server (left) and then with using the proxy server (right) side by side.
The website was able to easily identify my IP address in the first scenario, whereas this was not the case with the second one.
Congratulations ๐, you've just created your own proxy server, which hides your information from different servers. This is a basic implementation; you can add more features to make it more secure and manageable.
Taking Your Proxy Server to the Next Level
While your basic proxy server is functional, it lacks key security and management features. We will consider enhancing it with the following :
Logging: Implement logging to track requests, including user IP addresses, timestamps, and request details. This is crucial for monitoring and troubleshooting.
Rate Limiting: Add rate limiting to prevent abuse and ensure fair usage. This feature can limit the number of requests a user can make within a specified time frame.
Create a new model directory and add two new files to this directory logs.go and ratelimiter.go . Add the following code snippet to the respective files :
package model
import (
"fmt"
"os"
"time"
)
type Logs struct {
Date time.Time
ClientIp string
Method string
ServerAddress string
Agent string
}
func Log(clientIp, method, serverIp, agent string) Logs {
return Logs{
Date: time.Now(),
ClientIp: clientIp,
Method: method,
ServerAddress: serverIp,
Agent: agent,
}
}
var (
filename = "./logs.txt"
)
func (l *Logs) AppendLog() error {
flag := false
if _, err := os.Stat(filename); os.IsNotExist(err) {
flag = true
}
file, err := os.OpenFile(filename, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
return fmt.Errorf("failed to open log file: %w", err)
}
defer file.Close()
if flag {
_, err = fmt.Fprintf(file, "%s \t %s \t %s \t %s \t %s\n",
"Date", "Client-Ip", "Method", "Server-Address", "User-Agent")
if err != nil {
return fmt.Errorf("failed to write to log file: %w", err)
}
}
_, err = fmt.Fprintf(file, "%s \t %s \t %s \t %s \t %s\n",
l.Date.Format(time.RFC3339), l.ClientIp, l.Method, l.ServerAddress, l.Agent)
if err != nil {
return fmt.Errorf("failed to write to log file: %w", err)
}
return nil
}
The logs.go file defines a new Log model with fields such as the client's IP address, date, time, the domain being accessed, and the user agent. The AppendLog() function is responsible for appending logs to the log.txt file, if this file is not present it will automatically create one for you whenever the first successful request is made.
package model
import "time"
type RateLimiterModel struct {
Requests map[string]time.Time
}
var RateLimiter = &RateLimiterModel{
Requests: make(map[string]time.Time),
}
A RateLimiter is a map that stores the timestamp of the last request made by a user. It helps to determine if requests are being sent too frequently. This is important because frequent requests can indicate a potential attack, such as a Denial of Service (DoS) or Distributed Denial of Service (DDoS) attack.
In a DoS attack, a single source attempts to overwhelm a target server or network with a flood of requests, rendering it inaccessible to legitimate users. A RateLimiter can detect and prevent such attacks by limiting the rate at which requests are processed from a single source.
In a DDoS attack, multiple sources (often compromised computers or devices) send a flood of requests to a target server or network, overwhelming its capacity. A RateLimiter can help mitigate the impact of a DDoS attack by limiting the rate of requests from each source, reducing the overall volume of malicious traffic reaching the target.
To complete the setup, we need to integrate the logging and rate limiting features into our proxy server. This involves modifying the controller.go file to handle requests and implement these features. Here's how you can do it:
package controller
import (
"context"
"fmt"
"io"
"net"
"net/http"
"net/url"
"strings"
"time"
"github.com/Uttkarsh-Raj/Proxie/model"
"github.com/gin-gonic/gin"
)
func ProxyServer() gin.HandlerFunc {
return func(c *gin.Context) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*60)
defer cancel()
// Get the query URL
queryURL := c.Query("url")
if queryURL == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Missing query parameter 'url'"})
return
}
// Convert to a uri
targetUrl, err := url.Parse(queryURL)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error parsing target URL: %s", err)})
return
}
// Check the last requested time < 5sec
err = RateLimitChecker(c.ClientIP())
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
// Create a new client to send the request using this servers context
// Request the target url
client := &http.Client{}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, targetUrl.String(), c.Request.Body)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error creating request: %s", err)})
return
}
resp, err := client.Do(req)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error connecting to the destination server: %s", err)})
return
}
defer resp.Body.Close()
// Copy the response and the headers received
c.Status(resp.StatusCode)
for k, v := range resp.Header {
c.Header(k, v[0])
}
// Copy the response body to the current response
if _, err := io.Copy(c.Writer, resp.Body); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error connecting to the destination server: %s", err)})
return
}
newLog := model.Log(c.ClientIP(), resp.Request.Method, resp.Request.Host, c.Request.UserAgent())
err = newLog.AppendLog()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error logging the information: %s", err)})
return
}
model.RateLimiter.Requests[c.ClientIP()] = time.Now()
}
}
func RateLimitChecker(clientIP string) error {
if time.Since(model.RateLimiter.Requests[clientIP]) < time.Second*5 {
return fmt.Errorf("error: Please wait for %s seconds before next request", ((5 * time.Second) - (time.Since(model.RateLimiter.Requests[clientIP]).Abs().Round(time.Second))))
}
return nil
}
The first modification integrates the RateLimitChecker function, which examines the client's IP address to determine if it already exists in the map. If the IP address is found, the function calculates the time elapsed since the last request. If this duration is less than 5 seconds, it implies that the last request was made recently, and the client must wait at least 5 seconds before sending another request. If the duration exceeds 5 seconds, a nil error is returned. This leads to setting a new value for the client's IP address, indicating either a new address or that the last request was made more than 5 seconds ago.
The second change is the addition of the AppendLog function, which updates the log.txt file with information about the new request. With this, all the features have been implemented and are ready to use with a bit improved security and maintainability.
In the next section, we will write tests to validate the functionality of our proxy sever and ensure that it meets our requirements for performance, security, and reliability.
Integrating Tests for Proxy Server
Tests are crucial for ensuring the reliability and stability of our proxy server implementation. They help us verify that our server behaves as expected under different scenarios and conditions. Tests can ensure that the rate limiter functions correctly and the proxy server behaves appropriately when handling requests.
For the test code, create a file named main_test.go and add the following code to it:
package main
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/Uttkarsh-Raj/Proxie/controller"
"github.com/gin-gonic/gin"
"github.com/go-playground/assert/v2"
)
func TestProxyServer(t *testing.T) {
gin.SetMode(gin.TestMode)
router := gin.Default()
router.GET("/", controller.ProxyServer())
// Test the proxy server's function
req, _ := http.NewRequest("GET", "/?url=https://www.tsetit.com/", nil)
req.Header.Add("X-Forwarded-For", "123.123.123.123")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code)
expectedResponse := "<html><h1>Test Page</h1></html>"
assert.Equal(t, expectedResponse, w.Body.String())
// Test the rate limiter
err := `{"error":"error: Please wait for 5s seconds before next request"}`
req, _ = http.NewRequest("GET", "/?url=https://www.tsetit.com/", nil)
req.Header.Add("X-Forwarded-For", "123.123.123.123")
w = httptest.NewRecorder()
router.ServeHTTP(w, req)
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Equal(t, err, w.Body.String())
}
As you can see, the test is divided into two parts.
The first part tests the proxy service by sending a request to a test URL, https://www.tsetit.com/. Since our function handler makes requests to these URLs, we must ensure that this is not a valid URL and is only for testing purposes. The response collected is then tested against an expected result.
The second part tests the rate limiter. After the proxy test request is completed, a similar request is generated to test if the rate limiter is working correctly. This approach helps identify any issues in the code and ensures that the rate limiter is functioning as expected.
But running these tests now will still give us some error and will fail. We still need to make some changes in the controller.go file to be able to test the code. The final file should look like this:
package controller
import (
"context"
"fmt"
"io"
"net"
"net/http"
"net/url"
"strings"
"time"
"github.com/Uttkarsh-Raj/Proxie/model"
"github.com/gin-gonic/gin"
)
func ProxyServer() gin.HandlerFunc {
return func(c *gin.Context) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*60)
defer cancel()
// Get the query URL
queryURL := c.Query("url")
if queryURL == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Missing query parameter 'url'"})
return
}
// Convert to a uri
targetUrl, err := url.Parse(queryURL)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error parsing target URL: %s", err)})
return
}
// Check the last requested time < 5sec
err = RateLimitChecker(getClientIP(c))
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
// Create a new client to send the request using this servers context
// Request the target url
client := &http.Client{}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, targetUrl.String(), c.Request.Body)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error creating request: %s", err)})
return
}
// For Running Tests
if queryURL == "https://www.tsetit.com/" {
model.RateLimiter.Requests[getClientIP(c)] = time.Now()
c.Status(http.StatusOK)
_, _ = c.Writer.WriteString("<html><h1>Test Page</h1></html>")
return
}
resp, err := client.Do(req)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error connecting to the destination server: %s", err)})
return
}
defer resp.Body.Close()
// Copy the response and the headers received
c.Status(resp.StatusCode)
for k, v := range resp.Header {
c.Header(k, v[0])
}
// Copy the response body to the current response
if _, err := io.Copy(c.Writer, resp.Body); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error connecting to the destination server: %s", err)})
return
}
newLog := model.Log(c.ClientIP(), resp.Request.Method, resp.Request.Host, c.Request.UserAgent())
err = newLog.AppendLog()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Error logging the information: %s", err)})
return
}
model.RateLimiter.Requests[getClientIP(c)] = time.Now()
}
}
func RateLimitChecker(clientIP string) error {
if time.Since(model.RateLimiter.Requests[clientIP]) < time.Second*5 {
return fmt.Errorf("error: Please wait for %s seconds before next request", ((5 * time.Second) - (time.Since(model.RateLimiter.Requests[clientIP]).Abs().Round(time.Second))))
}
return nil
}
func getClientIP(c *gin.Context) string {
if c.Query("url") != "https://www.tsetit.com/" {
return c.ClientIP()
}
forwardHeader := c.Request.Header.Get("x-forwarded-for")
firstAddress := strings.Split(forwardHeader, ",")[0]
if net.ParseIP(strings.TrimSpace(firstAddress)) != nil {
return firstAddress
}
return getClientIP(c)
}
The addition involves a new getClientIP function, which will return the client IP in case of a normal request. However, when running tests, it will identify the URL and return the set IP address from the header for proper functioning. Replace this function with wherever you have used c.ClientIP() previously. To run these test you can use any of the following commands :
$ go test // Run tests in the current directory.
$ go test -v ./... // Runs tests in the current directory and all its subdirectories,
In Summary
In this blog, we've explored the world of proxy servers and learned how to create our own using the Go programming language. We started by understanding the basics of proxy servers and their importance in enhancing security and privacy online. We then delved into the implementation details, setting up a simple proxy server using Go's powerful tools and libraries. Along the way, we added features like logging and rate limiting to improve the server's functionality and security.
Building your own proxy server can be a rewarding experience, providing valuable insights. Whether you're looking to protect your online privacy or manage web traffic more efficiently, a custom proxy server can be a powerful tool in your toolkit.
Ready to take a peek? Head over to Proxie and explore the project. Your thoughts and contributions are super welcome. And if you find the project useful, consider giving it a star on GitHub.๐
GitHub: https://github.com/Uttkarsh-raj/
Twitter : Uttkarsh