░ ▓ ▒ ░ ░ ▒ ▓ ░
░ ▓ ░ ███▄ ░ ▓ ░
░ ██∙███ ░
█▄ ▄█│ ▄▄█████│ ░ ██ ██ ░ │███ │█████▄▄ ▀█ ▄
████│██▄██│ ██═════▀█│ █████│ ▀█▀█ ░│██ │█ │█▀═════██ │██ │█ │███▄
█│ █│█ █│██ ██│ █│ █▌ ▐█ │██ │██ ██│█│█│█│█▀ ▐▀
██▌ █│ █│██│ ██│ █│ █████ │██ │█ │██ │██│█│█│█│█▄
█│ █│ █│██│ ██│ ▐│ ███▀ │████ │██ │██│█ │██ │██▄
▀███│▀ █│ ██ ██│ ███ │██ ██ │█ │██ │█
▀ █████▀▀ ▌ ██ ▐ ▀▀█████ ▀ ▀██▀
░ ▄▄ ▄▀ ▀█ ▀▄ ▄▄ ░
░ ▓ ▀▀▀█▀ ▀ ▀█▀▀▀ ▓ ░
░ ▓ ▒ ▀▀▄▄▄▄▄ . EMOTICONS . ▄▄▄▄▄▀▀ ▒ ▓ ░
░ ▓ ▒ ▀▀▀▓███▓▄▄█▄▄▓███▓▀▀▀ ▒ ▓ ░
░ ▓ ▒ ▀▀▀▀█▀▀▀▀ ▒ ▓ ░
░ ▓ ▒ : : ▒ ▓ ░
░ ▓ ▒ :~~~~~~~~~~~~. PROUDLY PRESENTS .~~~~~~~~~~~~: ▒ ▓ ░
░ ▓ ▒ : : ▒ ▓ ░
░ ▓ ▒ emoticons-guides ▒ ▓ ░
▓ ▒ These are some guides that ▒ ▓
▒ are related to computer and ▒
interests.
Take a look around! :)
: :
:~~~~~~~~~~~~~~~~~. TAGS .~~~~~~~~~~~~~~~~~~~:
: :
Extension
Piracy
Privacy
RE
Steganography
Util
░ : : ░
░ ▓ :~~~~~~~~~~~~~~~. WRITTEN BY .~~~~~~~~~~~~~~~: ▓ ░
░ ▓ ▒ : : ▒ ▓ ░
░ ▓ ▒ /u/emoticons ▒ ▓ ░
░ ▓ ▒ : : ▒ ▓ ░
░ ▓ ▒ :~~~~~~~~~~~~~~~. DISCLAIMER .~~~~~~~~~~~~~~~: ▒ ▓ ░
░ ▓ ▒ : : ▒ ▓ ░
░ ▓ ▒ emoticons-guides is very basic information ▒ ▓ ░
░ ▓ ▒ and maybe not all accurate ▒ ▓ ░
-
Examining the ZIP File format [Steganography] [Privacy] (TODO)
-
Examining the PNG File format [Steganography] [Privacy] (TODO)
TLS Record Protocol
it is used to verify the identity of servers with asymmetric key and to make a symmetric key exchange from master secrets. The master secret is split up into keys, MAC keys and if require IV keys for the client and server both. (example: using diffie hellman from client and server shared pre master secrets) bytes are big endian. also can do Alert Protocol and Change Cipher Spec Protocol. As of this day, TLS 1.3 is the most up to date of this protocol. TLS 1.3 simplifies handshake by a lot. TLS comes from SSL, but TLS is more better with more update encryption.
The Record
The record just a wrapper for TLSCompressed (same format as TLSPlaintext) structure, or a TLSCiphertext. the first byte shows the content type. The next byte show the major protocol version. the next byte show the minor protocol version. The next two show the length. Then follows the data in the record. this data is compressed using the algorithm from the handshake, but on starting handshake, there is no compression. Encryption is done after compression, depending on cipher spec. MAC is added to compressed records. MAC algorithm depend on cipher spec. MAC will be calculated before encryption. A record can have more than one message of the same content type or the partial content of single message.
Handshake TLS1.2
A handshake is done from a client first. and a handshake done on top of the record.
- ClientHello and ServerHello
Client and server decide on the TLS version, cipher suite and compression used. client and server share random values. a session identify assigns to the client if not connection before. this is always a random session identity. If a client shows a previous session identifier to the server, it can continue a session because the cryptography can be expensive, a session will make sure the key exchanges only done once.
- Server Certificate (optional)
server will send intermediate and its own certificate to verify identity. Intermediate can be zero or more. Can send the root certificate as well, but is not required. Server certificate will contain a server public key
- Server ServerKeyExchange (optional)
server sending a public key used to make the pre master secret. Only sent if server has no certificate or server's certificate is used for only signing
- Server CertificateRequest (optional)
If the server need to confirm client identity.
- Server ServerHelloDone
This tells client that server hello and server part of key exchange is finish
- Client Certificate (optional)
Client will send certificate to verify identity. Only if request by Server CertificateRequest. if client has no certificate, message sent with zero length in data. Certificate or certificate chain sent the same as if a server sending the chain. client certificate contains client public key
- Client ClientKeyExchange
for rsa client sends a pre master secret, encrypted in server certificate public key.
- Client CertificateVerify (optional)
If a client send a certificate, it is verified by the digital signature. Indicating the client does has the private key for its certificate. Only send if the certificate is not have fixed Diffie Hellman. a hash of all prior handshake message is sent
- Client ChangeCipherSpec
uses a different content type in the record. notify server that next messages will compress and encrypt with the chosen cipher suite and compression
- Client Finished
message encrypted with algorithms in the cipher spec. contains HMAC number generated from MD5 and SHA1 from master secret, "client finished" string, and hash of all prior handshake message
- Server ChangeCipherSpec
Using a different content type in the record. notify client that next messages will compress and encrypt with the chosen cipher suite and compression
- Server Finished
message encrypted with algorithms in the cipher spec. contains HMAC number generated from MD5 and SHA1 from master secret, "server finished" string, and hash of all prior handshake message
Examining TLS ClientHello [Privacy] (TODO)
In the TLS record protocol, a ClientHello message sent to the server before connecting. the message get sent unencrypted because this message and other in the handshake are used to get the encryption and compression method for future communication.
This message can be used to fingerprint your device and browser, even before sending the first HTTP request. Some data like the TLS record version, cipher suite list, compression method list, and Hello Extensions that are used or not may be unique for different browser or software on different device. Which allow for fingerprinting.
(todo: write a detailed explanation about how this is done)
Additional links
The Secure Sockets Layer (SSL) Protocol Version 3.0: https://datatracker.ietf.org/doc/html/rfc6101
The Transport Layer Security (TLS) Protocol Version 1.2: https://datatracker.ietf.org/doc/html/rfc5246
The Transport Layer Security (TLS) Protocol Version 1.3: https://datatracker.ietf.org/doc/html/rfc8446
Information missing here
Alert Protocol
Initialization Vector
Stream Ciphers
Block Ciphers
AEAD Ciphers
HTTP/1.1 Webclient [Util]
How to request to server with HTTP/1.1
A request pipeline can be done by having separate thread for both sending and receving from the socket.
Common headers
Higher q value is higher preference for an option. q value can be from 0 to 1 and use 1 to 3 decimal places. The q value only applies to the preceding option
Accept
mime types that receiver expecting. comma separates, semicolon for q value. Implied q value is 1.0. Mime type can be */* (any), type/* (any subtype of type) or type/subtype. More specific mime type with same q values are preceding less specific types with same q value.
Accept-Encoding
Compressions that receiver can use, but not necessary. only compression are br, compress, deflate, gzip, identity, and * (any) Comma separates, semicolon for q value. Implied q value is 1.0.
Accept-Language
language requested. comma separated, * for any, semicolon for q value. implied q value is 1.0
Authorization
Basic with base64 username:password. This gets sent when a url has the protocol://username:password@domain:port/ or when authorization is required from a response, but server may not use basic in that case. other authorization exist. port is optional and is default of 80 for http or 443 for https
Connection
close, keep-alive, or upgrade. HTTP/1.1 Connection
header default is keep-alive
compared to HTTP/1.0 which is close
. keep-alive will reuse the socket for requests. close will close the socket once the request is done
Content-Length
length of body of request or response
Content-Type
mime type of body of request or response
Cookie
cookie=value; separated by semicolon
Host
Server domain or IP and optional port if port not 80 or 443. can be used for a proxy service
If-Modified-Since
check if modified since a date.
If-None-Match
check ETag for cache. can be multiple ETag with comma separating
Origin
Domain of current page.
Referer
full path of current page
Upgrade
The new protocol request by the client, multiple protocols can be comma separated. Server may reject
User-Agent
Browser version and maybe operating system. Can also have webrenderer
Common methods
GET
receive a resource. Should not modify state of resource on server. do not include sensitive data in the query, for that you need POST
HEAD
Perform like a GET request, but server will only send headers. Useful for ETag check
POST
for forms or for send large data, like a file with multipart/form-data mimetype. may change the resource. forms use application/x-www-form-urlencoded if not specified
OPTIONS
Used to determine the request methods that the server support. also be used to query the access control for cross origin requesting
Sample requests
GET / HTTP/1.1\r\n
Host: example.com\r\n
\r\n
simple GET request, using Host header. Host header must be present in all requests. Host header shows the domain example.com and no port, which means the port should be 80 if using http or 443 if using https. A GET request will not have a body, so Content-Length: 0
is not needed.
GET /index.php?show_page=100 HTTP/1.1\r\n
Accept: */*\r\n
Accept-Encoding: compress, gzip\r\n
Content-Length: 0\r\n
Host: example.com:1111\r\n
If-None-Match: "05aafa1b"\r\n
User-Agent: Mozilla/5.0 (OS 1.0)\r\n
\r\n
A request with a query. The file is index.php and query is show_page=100
. Accept: */*
means the client will accept any mime type response from the server. the encoding the client wants is compress or gzip, with both implied q=1.0 which is equal preference, but the server does not need to compress. Host: example.com:1111
shows the domain example.com and port 1111. If-None-Match
will use an ETag for the page to only get the page when the ETag is not found on the server for the request page.
POST / HTTP/1.1\r\n
Accept: text/html, application/json; q=0.5\r\n
Accept-Language: zh, de-DE, *; q=0.575\r\n
Content-Length: 9\r\n
Content-Type: application/x-www-form-urlencoded\r\n
Host: example.com\r\n
\r\n
form=data
A POST request with some form data. The form uses POST method and post to /
. Accept
has text/html with implied q=1.0 and application/json with q=0.5, which mean text/html has a higher preference. Accept-Language
has zh and de-DE languages both with implied q=1.0 equal preference. zh and de are base language. A sub language (ex: DE) can come after base language, but it is optional. zh language has no sub language in this request. *; q=0.575
means all other languages have a q value of 0.575 and is an example of a q value with 3 decimal places which is the maximum of decimal places. Content-Length
now is the number of bytes in the body. the Content-Type
is the default for forms. In the body of the request, the form field with the name 'form' has the value 'data'.
POST /?a=1&b=2 HTTP/1.1\r\n
Content-Type: multipart/form-data; boundary=boundary\r\n
Host: example.com\r\n
\r\n
--boundary\r\n
Content-Disposition: form-data; name="text"\r\n
\r\n
value\r\n
--boundary\r\n
Content-Disposition: form-data; name="file"; filename="file"\r\n
\r\n
file... (-snip-)
\r\n
--boundary--
A POST request with some form data and a query. This form also uses POST method and posts to /
with the query a=1
and b=2
. Queries are separated with &
and are encoded with percent signs and hexadecimal for any characters like spaces, ?
, =
, &
and others. this Content-Type
uses multipart/form-data
with a boundary of boundary
, so the text --boundary--
must not appear after a CRLF in any of the data except at end. The boundary may be in quote or not. In the body of the request, the boundary after --
is there and then CRLF. Then Content-Disposition
header is used for data about the form field. other headers may follow this. It ends with CRLFCRLF like the request headers. Then the value with another CRLF which is not part of the value. if the request body is done, a --
is put and then the boundary and then another --
. If not done, then the same process repeats.
This body shows a form with a text field with the name text
and the value value
and a file field with the name file
which has filename file
on the disk and then data of the file.
Additional links
Hypertext Transfer Protocol -- HTTP/1.1: https://datatracker.ietf.org/doc/html/rfc2616
Information missing here
Range requests
Other Authorization
Other headers
HTTP/1.1 Webserver [Util]
Sample response and header information. same pipeline as client can be done
Common headers
Content-Encoding
If compression applied. multiple can be applied, separate by comma in order of the applied. Only compression are br, compress, deflate, and gzip.
Content-Type
Same meaning as client.
Content-Length
Same meaning as client. does not appear if Transfer-Encoding: chunked
Connection
Same meaning as client.
Date
The date and time in GMT.
ETag
value to indicate to client if modified or not
Location
the next url to go to for redirects. Starts on current path, unless begins with forward slash or full url
Server
description of server and version.
Set-Cookie
Set a cookie=value. Can limit to a domain or path. Can set to expire or force expire if date is before current date. Can be httponly (javascript can not use the cookie).
Transfer-Encoding
If compression applied or if chunked. both can apply, separate with comma. Allowed from chunked, compress, deflate, gzip, and identity. chunked uses hexadecimal number as size of the chunk, follow by CRLF, then chunk data, then CRLF. chunked ends when 0 length chunk followed by CRLFCRLF. Headers may come after the zero length chunk after the first CRLF. Transfer-Encoding: chunked
must be the last thing done to the body
Common statuses
101 Switching Protocols
200 OK
206 Partial Content
301 Moved Permanently
302 Found
304 Not Modified
403 Forbidden
404 Not Found
500 Internal Server Error
Sample responses
HTTP/1.1 101 Switching Protocols\r\n
Connection: upgrade\r\n
Upgrade: HTTP/2.0\r\n
\r\n
When the server accepts the client request to switch protocols or the server wants to upgrade to process the request.
HTTP/1.1 200 OK\r\n
Connection: keep-alive\r\n
Content-Length: 100\r\n
Content-Type: text/html; charset=utf-8\r\n
Date: Mon, 05 Jul 2021 00:43:22 GMT\r\n
ETag: W/"1111"\r\n
Server: server/1.2.3 (Unix)\r\n
Set-Cookie: cookie=value\r\n
\r\n
<!DOCTYPE HTML><html>. . . (-snip-). . . .</html>
A response with a 100 byte html in utf-8 encoding. the connection will stay open when after send to client. The Date
header shows the date. the ETag
header gives the client a value to check if page change. The Server
header shows details about the server. the Set-Cookie
header wants the client to set the cookie cookie
with the value value
and is not httponly.
HTTP/1.1 200 OK\r\n
Content-Type: application/json\r\n
Transfer-Encoding: chunked\r\n
\r\n
6\r\n
{"mani\r\n
E\r\n
fest_version":\r\n
2\r\n
4}\r\n
0\r\n
\r\n
a application/json
response with unknown charset that is sent using Transfer-Encoding: chunked
. The Connection: keep-alive
is missing, but the connection will stay open because of HTTP/1.1. The first chunk is 6 bytes, the second chunk is 14 bytes, the third thunk is 2 bytes, and the last chunk is 0 bytes which meanings no more chunks is in the response.
HTTP/1.1 301 Moved Permanently\r\n
Connection: close\r\n
Location: /index.html\r\n
\r\n
A response that will redirect the client to /index.html
on the same domain and close the connection.
Additional links
Hypertext Transfer Protocol -- HTTP/1.1: https://datatracker.ietf.org/doc/html/rfc2616
Information missing here
HSTS (Strict-Transport-Security)
CSP (Content-Security-Policy)
CORS (Access-Control-Allow-Origin, Access-Control-Allow-Methods, ...)
Other headers
Other statuses
418 I'm a teapot (https://datatracker.ietf.org/doc/html/rfc2324#section-2.3.2)
HTTP proxy service [Piracy]
Once you have the webclient and webserver, you can combine them to make a proxy service. Proxy services are great for piracy because then you are able to intercept the very HTTP traffic and then do analyse on them to save specific parts to a file (ex: a video on the web loads m3u8, the proxy can capture that and save files if the links in the m3u8 go through the proxy. For encrypted video stream in the body of HTTP decrypted by the client, the HTTP proxy will be of no use unless you wish to document the encrypted stream on the file system. this does not mean the idea of proxy is dead, because you just need to proxy the decryption algorithm instead, which may use a different protocol and may be very hard depending if hardware is involved)
/etc/hosts
Using /etc/hosts can make the request go to 127.0.0.1 which the proxy can listen on. Windows use %windir%\System32\drivers\etc\hosts
Example
127.0.0.1 example.com
in /etc/hosts will make example.com domain go to 127.0.0.1
Sample proxy
proxy = tcp stream socket listening on port 80
client = socket from proxy accept
server = tcp stream socket
connect server to example.com on port 80
receive from client and send to server
receive from server and send to client
Using a thread pool, you can accept many client requests instead of one at a time. Also, make sure to check if HTTP/1.0 or HTTP/1.1 and the Connection
header to see if the socket should close. HTTP/1.0 closes if no Connection
header. HTTP/1.1 do keep-alive if no Connection
header.
Extension
If you wanting to proxy multiple domains, add them to the /etc/hosts. Change the proxy so that it doesnt connect to the server until the client is done with HTTP headers. With http headers copy, you can figure out the server that you should connect to in Host
header. send the copy and then continue receive from client and send to server.
Using openssl [Util]
Generate an RSA private key
$ openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out rsa.key
(sidenote: rsa2048 is default)
if you wanting RSA1024 or RSA4096, change the rsa_keygen_bits:2048
to rsa_keygen_bits:1024
or rsa_keygen_bits:4096
OR use
$ openssl genrsa -out rsa.key 2048
replace 2048 with number of bit for the key
To encrypt the private key file with a password make a file called password
in the directory and edit it so that it is the password on one line and ends with new line
$ openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out rsa.key -aes256 -pass file:password
OR use
$ openssl genrsa -out rsa.key -aes256 -passout file:password 2048
Replace aes256 with the cipher wanted to encrypt the key with
to view the key
$ openssl pkey -in rsa.key -text -noout
or the public part only
$ openssl pkey -in rsa.key -text_pub -noout
To view the key with password
$ openssl pkey -in rsa.key -passin file:password -text -noout
or the public part only
$ openssl pkey -in rsa.key -passin file:password -text_pub -noout
Generate an X509 root certificate
all root certs are self signed
First you need a certificate request. use private key for -key
$ openssl req -new -key rsa.key -out cert.req
to view
$ openssl req -in cert.req -text -noout
Then using the cert request, do
$ openssl x509 -req -in cert.req -signkey rsa.key -out root.cert
then, to view the cert, use
$ openssl x509 -in root.cert -text -noout
Generate a X509v3 root certificate
To do this, you need to specify the extensions in the cnf file. You can copy the authority key id from the signer certificate. the serial number of X509 certificate is arbitrary and chosen by the certificate authority.
A better way to generate the X509 root certificate
$ openssl req -new -x509 -key rsa.key -out root.cert
will generate a X509 root certificate with x509v3 extensions but cnf file is needed to control the extensions
sample req config
[req]
default_bits=2048
default_keyfile=default.key
distinguished_name=dn
attributes=req_attributes
prompt=no
req_extensions=v3
[req_attributes]
[dn]
C=NZ
ST=State
L=Locality
O=Organization
OU=Organizational Unit
CN=Common Name
[email protected]
[v3]
subjectKeyIdentifier=hash
basicConstraints=critical,CA:true
C is the 2 letters Country Code (ex: NZ), default.key file do not need to exist. only country name and common name are required in the dn section. Others like state, locality, etc are optionals
The subject key identifier can be set using a hex string if wanted
subjectKeyIdentifier="111122223333444455556666777788889999AAAA"
generate the certificate request
$ openssl req -new -key rsa.key -config config.cnf -out cert.req
Sample ca config
[ca]
default_ca=CA_default
[CA_default]
dir=.
database=$dir/index.txt
new_certs_dir=$dir/newcerts
certificate=$dir/ca.cert
serial=$dir/serial
private_key=$dir/ca.key
default_startdate=YYMMDDHHMMSSZ
default_enddate=YYMMDDHHMMSSZ
default_md=sha256
policy=policy_any
x509_extensions=v3
[policy_any]
countryName=supplied
stateOrProvinceName=optional
organizationName=optional
organizationalUnitName=optional
commonName=supplied
emailAddress=optional
[v3]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid
basicConstraints=critical,CA:true
Create a empty file called index.txt and make a directory called newcerts. ca.cert and ca.key should be the certificate and private key for the ca. ca.key can be rsa.key for this because of self sign. Create the serial file and put in "02" or a hex like "111122223333444455556666777788889999AAAA" or add rand_serial=yes
after the serial=$dir/serial
line in cnf for a random serial which will not use serial file. but the serial file must exist even if not used by rand_serial=yes
. default_md can be anything from the $ openssl dgst -list
Change the default_startdate and default_enddate (ex: 210731123456Z is 2021-07-31 12:34:56 GMT)
then make the cert
$ openssl ca -selfsign -key rsa.key -in cert.req -config config.cnf -out root.cert
X509v3 Extensions
Edit the v3 section
Sample Basic Constraints
basicConstraints=CA:true
basicConstraints=CA:false
basicConstraints=critical,CA:true,pathlen:0
pathlen is the number of intermediate certificate that can be in the chain after this certificate. If it is 0 the certificate can only sign the server certificate
Sample Key Usage
can be any comma separated values from digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign, cRLSign, encipherOnly, decipherOnly. Can have critical
keyUsage=digitalSignature
keyUsage=critical,dataEncipherment,cRLSign
Sample Extended Key Usage
can be any comma separated values from serverAuth, clientAuth, codeSigning, emailProtection, timeStamping, OCSPSigning, ipsecIKE, msCodeInd, msCodeCom, msCTLSign, msEFS. Can have critical
extendedKeyUsage=emailProtection
extendedKeyUsage=serverAuth,clientAuth
extendedKeyUsage=critical,serverAuth,clientAuth
Sample Certifiate Policies
can be any comma separated values of policy oid. Can have policy sections but not shown here
certificatePolicies=1.2.3.4
certificatePolicies=1.2.3.4,1.1.1.1.1
Sample Alternate DNS
subjectAltName=@alt_names
[alt_names]
DNS.1=127.0.0.1
DNS.2=example.com
DNS.3=*.example.com
Generate a X509 certificate chain
In order to verify a certificate, the hash of the certificate is taken and signed by the signer cert's private key. using the same hash algorithm, and the public key of the signer cert, it checks if the signature matches (same hash).
if previous step from the guide was done, rename rsa.key
to root.key
. If not, this step needs a file called root.cert
and root.key
which is the root self sign certificate and the root private key
Generate private keys called intermediate.key and server.key and make requests for them. Make distinguished_name section something else for each request. for server.req use common name for domain name or use alternate dns extension. To create intermediate.cert, change the certificate=$dir/ca.cert
to certificate=$dir/root.cert
and change private_key=$dir/ca.key
to private_key=$dir/root.key
. change the dates and default_md and extensions if desire then
$ openssl ca -in intermediate.req -config config.cnf -out intermediate.cert
Then change certificate=$dir/root.cert
to certificate=$dir/intermediate.cert
and change private_key=$dir/root.key
to private_key=$dir/intermediate.key
. change the dates and default_md and extensions if desire then
$ openssl ca -in server.req -config config.cnf -out server.cert
Congratulations you now have a certificate chain!
Additional links
genpkey - generate a private key: https://man.archlinux.org/man/genpkey.1ssl
pkey - public or private key processing tool: https://man.archlinux.org/man/pkey.1ssl
config - OpenSSL CONF library configuration files: https://man.archlinux.org/man/config.5ssl.en
req - PKCS#10 certificate request and certificate generating utility: https://man.archlinux.org/man/req.1ssl
x509 - Certificate display and signing utility: https://man.archlinux.org/man/x509.1ssl
x509v3_config - X509 V3 certificate extension configuration format: https://man.archlinux.org/man/core/openssl/x509v3_config.5ssl.en
ca - sample minimal CA application: https://man.archlinux.org/man/ca.1ssl
Information missing here
Other Certificate Policy
Other Subject Alternative Names
CRL
CTL
SCT
OCSP
Related
LibreSSL
GnuPG
GnuTLS
HTTPS proxy service [Piracy]
Using information from openssl to create certificate, you only need to change sockets to ssl wrapped sockets which have a context of your certificate chain and server private key. This is really easy if you already have your code for your own HTTP proxy service. The ssl contexts will do the encryption. Remember to install your proxy's root certificate in both the certificate store of the ssl context and the client.
Sample proxy
context = new ssl context
give this ssl context your server certificate private key
give this ssl context your server certificate chain or
the root certificate if there is no chain and the common
name is domain
defaultContext = default ssl context
proxy = tcp stream socket
wrap proxy socket in the ssl context then
make proxy listen on port 443
client = socket from proxy accept
server = tcp stream socket
wrap server in the default ssl context then
connect server to example.com on port 443
receive from client and send to server
receive from server and send to client
Same advice and extension as the HTTP proxy, but just use ssl contexts. To proxy multiple domains, it may be needed to generate many contexts and certificates or use alternate DNS name with a single certificate.
HTTP/2 [Util] [Extension] (TODO)
HTTP/2 is released on 2015. It is the same fundamental design as HTTP/1.1 (use tcp/ip and tls, but for HTTP/2 tls is like a requirement and not optional). HTTP/2 no longer uses request pipeline as HTTP/1.1 (but replaces with multiplexing). HTTP/2 marks Transfer-Encoding: chunked
HTTP header invalid. To upgrade to HTTP/2 a client sends header with Connection: upgrade
and Upgrade: HTTP/2.0
(todo: explain HTTP/2 frames)
HTTP/3 [Util] [Extension] (TODO)
HTTP/3 is released on 2019. HTTP/3 uses HTTP/2 api, but over udp instead of tcp. They made new protocol called QUIC which pushes packet resending and integrity information to application level instead of kernel level. This eliminates head of line blocking in HTTP/2 (todo: explain QUIC)
WebSockets (TODO)
The client will send a HTTP request with Connection: upgrade
and Upgrade: websocket
(todo: examine the protocol and see if it can be used to intercept media)
Insecure WebSockets (TODO)
(todo: examine the protcol)
Secure WebSockets (TODO)
secure websockets operate on TLS (todo: examine the protcol)
Additional links
The WebSocket Protocol: https://datatracker.ietf.org/doc/html/rfc6455
Examining DirectShow WindowsAPI [Privacy] (TODO)
(todo: examine if it is possible to create a fake webcam feed using C++ and DirectShow WinAPI for privacy reasons)
Examining the ZIP File format [Steganography] [Privacy] (TODO)
The zip file format has an extra field in local file headers that may be used for metadata. the zip file format version made by can fingerprint (ex: 0x1e03 in the version made by is a little endian 2 byte integer specifying that the zip file was created on *nix and the version support 3.0 of the zip standard). the external file attributes may be used for fingerprinting.
Because the central directory header of a zip file is at the end, it easy to append a zip file to like an image file (very basic steganography). (todo: how to change file creation time using a zip file) (todo: more details)
Additional links
.ZIP File Format Specification 6.3.3: https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.3.TXT
Examining the PNG File format [Steganography] [Privacy] (TODO)
The png starts with hex 89504E470D0A1A0A and then png chunks come after. integers are big endian. Each chunk contains the 4 byte unsigned integer for the length of the data, the chunk type is the 4 byte ascii for the chunk type, the data, and then the CRC check for the data includes chunk type and chunk data but not the length. Each 4 byte ascii can be checked with bitwise and of 32 (&32) to make additional info about headers.
Because of ascii, if it lowercase, then the bitwise and is 32 and 0 for uppercase. if the first letter is uppercase, then the chunk is required in the image. it is lowercase then the chunk is not. If the second letter is uppercase, then the chunk type is public. If it is lowercase then the chunk type is private. the third letter case is reserved. if the fourth letter is uppercase, then the chunk should not be copy and if lowercase the chunk can be copy.
The CRC is done using the number EDB88320 for table calculation (read this hexadecimal number in binary from right to left to get the polynomial but without 32 power)
IHDR must be first chunk and IEND must be the last chunk. for the simplest image, one or more IDAT come after IHDR and those are the only 3 chunk type in the image. all IDAT are compressed using deflate
IHDR has a length of 13 and add chunk type ascii for hex 0000000D49484452. The first 4 byte are the image width, the next 4 bytes is the image height. The 5 next are all 1 byte which are the bit depth, the color type, the compression method, the filter method, and interlace method.
The example image is using bit depth 8, color type 6 (see below), compression method 0 (deflate), filter method 0 (null), interlace method 0 (null).
There are only 5 type of png image
Truecolor and alpha = 6
image has RGB values and alpha value
Grayscale and alpha = 4
image has gray value and alpha value
Truecolor = 2
image has RGB values
Grayscale = 0
image has gray value
Indexed = 3
image use a table to index at most 256 distinct colors. this table must be the PLTE chunk and only appear before any IDAT
Every pixel is made from channels with a sample depth. In the png, the channels have the same sample depth. The pixels come from samples of the channels or the index if indexed color. The bit depth represents the number of bits per sample or the bits in every table color if indexed color.
the image can use Adam7 interlace which make different images from the original pixels for each group of pixel in a pattern, each scanline is made from the pattern and represent that pixel group. Then all scanlines from top to bottom are turned into bytes, with the left pixel first and the right pixel last. it can use filter before compression to help the deflate compress. Then it is compressed stream and placed into image data chunks.
after the IHDR CRC comes the IDAT chunk hex ????????49444154. replace ? with length of chunk in hex big endian. Some programs limit IDAT length. Inside the IHDR chunk is deflate compressed data. The first byte is the compression method which is 08 (deflate), the next byte is the flags which is 1D (00011101 in binary). The first two bits 00 indicates compression with the fastest algorithm. the next bit 0 indicates no dictionary. The next bits 11101 are the check value for the two bytes where the first byte * 256 + the entire flags byte must be multiple of 31.
8*256 + 29 = 2048 + 29 = 2077 which is multiple of 31
then the compressed data is next. The byte will be the deflate method header and it 01 (binary 00000001). because last bit is 1, this is the last compressed block. The two bits to left of the 1 are the compression type which is 00 for no compression. other compression types are 01 for fixed Huffman, 10 for dynamic Huffman. 11 is error. Because of compression type no compression, next two byte is the length. This is the first byte + 256 * second byte. then next two byte is the one complement of the length.
After this is the actual compressed image data (but this using type 00 for no compression) which mean only length from deflate header bytes after that. Then is the ADLER-32 of the uncompressed data. Then is the CRC of the image chunk.
How compute the ADLER-32 can be found in the RFC1950 and CRC32 can be found in the W3 specification
After this a 0 length IEND header 0000000049454E44AE426082 ends the image.
Steganography
-
The lower bit of selection of pixel sample can be change to make a binary message in the file which wont be visible to a human eye because it only affect lower bit. but only do this to image that is colorful
-
image is truecolor with alpha (type 6) and since the bit depth is 8, it fits ascii in the RGBA pattern. If G and A are 0, it is like wide string where R and B are hold the ascii. Because A are 0 in every pixel, the image look transparent but actual contains some data
download the image as image.png and try using strings
in linux or open with notepad on windows
$ strings image.png
$ strings -e l image.png
notice the different result between the outputs.
-
Based on image chunks and not pixels there are still ways to do steganography. Example: send a lot of png, but make the number of IDAT chunks mean something. or use the PLTE chunk and other metadata chunks since it can has lots of bytes. Or make the specific chunk ordering different. there are a lot of way to do this based on only chunk alone
-
Or use pixels and chunks both for the steganography
Fingerprinting
It may be possible but very unlikely (i think) to discover the program used to make the picture by checking chunk ordering, chunk present and chunk count (ex: some program may include sRGB and gAMA chunk while other does not), private chunks, how and when the compression is done, color palette, color type, or other chunk metadata
some chunk like a text chunk can have metadata or compressed metadata for a keyword. Even if program that made picture is not know, this text metadata content may fingerprint.
Additional links
Portable Network Graphics (PNG) Specification (Second Edition): https://www.w3.org/TR/PNG/
ZLIB Compressed Data Format Specification version 3.3: https://datatracker.ietf.org/doc/html/rfc1950
Adam7 algorithm: https://en.wikipedia.org/wiki/Adam7_algorithm
Information missing here
Interlaced image
Other chunks (cHRM, gAMA, iCCP, sBIT, sRGB, bKGD, hIST, tRNS, pHYs, sPLT, tIME, iTXt, tEXt, zTXt) and their purpose and order in file
Filtered scanline functions
Changing File times [Privacy] (TODO)
Windows (TODO)
The creation time can be changed using a zip file (see: Examining the ZIP File format (TODO)), but you need to use MSDOS time and date (both 16bit integer) (todo: what is MSDOS time and date)
Otherwise, you need the SystemTimeToFileTime
and SetFileTime
functions of the windows api
Linux
You can use stat
to see the info for a file
$ stat somefile
File: somefile
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 801h/2049d Inode: 6160770 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ user) Gid: ( 1000/ user)
Access: 2021-07-14 06:38:12.138649334 +0000
Modify: 2021-07-14 06:38:12.138649334 +0000
Change: 2021-07-14 06:38:12.138649334 +0000
Birth: -
we are interested in the access, modify, and change times
You can use touch
to set time info for a file. you can only have full control over access and modify. change times are updated by the kernel to the current time. touch -a
to change access. touch -m
to change modify.
$ touch -a -d "2000-08-08 12:34:56.1234123" somefile
$ touch -m -d "2008-04-09 11:22:33.4444123" somefile
$ stat somefile
File: somefile
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 801h/2049d Inode: 6160770 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ user) Gid: ( 1000/ user)
Access: 2000-08-08 12:34:56.123412300 +0000
Modify: 2008-04-09 11:22:33.444412300 +0000
Change: 2021-07-14 06:49:42.294649334 +0000
Birth: -
The resolution of time may not be as precise because of the used file system.
(i do not recommend) to change the change time, you need to either patch the kernel for your system to stop automatically setting the change time, or you need to open the entire disk at a very low level and find and change data of the file's inode (none of these options are recommend by me!)
DEFLATE [Util] (TODO)
Huffman codes are prefix codes which mean that it's a set with no element being a prefix of any other. This variable length coding can be done based on frequency, which means that the shorter lengths get assigned to characters that appear more often. there is also backreference and length pairs that can repeat strings up to 32K before in the buffer. The Huffman tree can be reconstructed from a set of integers that are the bit lengths and count of bit lengths in a lexicographic ordering. (todo: describe the compression algorithm)
Additional links
DEFLATE Compressed Data Format Specification version 1.3: https://datatracker.ietf.org/doc/html/rfc1951
Make a Firefox Addon [Util]
A Firefox addon can useful to add javascript into websites or to intercept HTTP requests. Also, there are many other types of Firefox addons to make. for each of the addons in the guide, make a directory if seems useful.
Content Scripts
This addon will add javascript into websites.
File: manifest.json
{"manifest_version":2,
"name":"the name",
"version":"1.0",
"description":"the description",
"icons":{"32":"image.png"},
"content_scripts":[{
"matches":["https://example.com/"],
"js":["script.js"]
}]}
File: image.png
32x32 png image
File: script.js
console.log('this is script')
Now install temporary addon and then go to https://example.com and open console. Try changing matches array in manifest.json by adding new urls or using wildcard like https://example.com/*
or *://*/*
. then see what happens when you visit urls. Change the script.js file to any javascript.
Intercept HTTP
This addon will intercept HTTP.
File: manifest.json
{"manifest_version":2,
"name":"the name",
"version":"1.0",
"description":"the description",
"icons":{"32":"image.png"},
"permissions":["*://*/*","webRequest","webRequestBlocking"],
"background":{"scripts":["script.js"]}}
(sidenote: you can both use content_scripts
and background
in manifest.json)
File: image.png
32x32 png image
File: script.js
function beforeRequest(detail){
if(!detail.url.indexOf('https://example.net')){
let f=browser.webRequest.filterResponseData(detail.requestId),
t=new TextEncoder();f.onstart=function(e){
f.write(t.encode('overwritten'));
f.close();f.disconnect()};
}
if(!detail.url.indexOf('https://example.com'))return{cancel:true};
return{};
}
browser.webRequest.onBeforeRequest.addListener(
beforeRequest,{urls:['*://*/*']},['blocking']);
Install temporary addon then this script will cancel all requests starting with https://example.com which can see if open console then go to network tab. Also will overwrite all requests start with https://example.net with text 'overwritten', try to open a new tab and go to https://example.com and then try https://example.net to see
Install Temporary Addon
Visit about:debugging then go to This Firefox link. Then load the addon by selecting any file (ex: manifest.json) in the addon directory. if using private mode, remember to visit about:addons to enable it for private mode
Additional links
Browser Extensions - Mozilla | MDN: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions
webRequest.onBeforeRequest - Mozilla | MDN: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onBeforeRequest
webRequest.StreamFilter - Mozilla | MDN: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/StreamFilter
Crack your First Software [RE] (TODO)
some tools: IDA, radare2 (linux), gdb (linux), Hopper, ghidra, ollydbg (windows, obsolete), x64dbg, windbg (todo: find software on web that would be easy for a beginner. go through thoughts and explain what to do to crack the software) (note: this is for beginner, do not discuss crack DRM here)
Cracking Java Software [RE] (TODO)
some tools: jdgui, java decompiler (todo: find software on web that would be easy for beginning)
Cracking .NET Software [RE] (TODO)
some tools: ILSpy (see source), de4dot (deobfuscate) (todo: get software on web that would be easy for beginner)
Making your First keygen [RE] (TODO)
(todo: find easy software to make a keygen for, then describe thought process and how it is can be done)
Cracking DRM [RE] (TODO)
some tools: DeDRM
handbrake with some libraries can also do it
Always online DRM can be bypassed if a server is programmed for it. /etc/hosts will be useful like in HTTP or HTTPS proxy
(todo: more research and aggregate and post back here)