Introduction:

Introducing a new component, such as an LLM Gateway, into your existing architecture prompts careful consideration. One key concern is minimizing dependencies and retaining compatibility with your current setup. Ideally, you aim to implement the LLM Gateway with minimal alterations to your LLM API integration code. This means continuing to utilize the native SDKs or libraries provided by your LLM API vendor, ensuring a smooth transition and preserving the integrity of your existing infrastructure.

In this article we explore how to use native functions of AWS Python SDK to make calls via LLM Gateway Gecholog.ai. AWS API calls are particually interesting due to the pre-signature process used by default by AWS.

AWS Pre-signature process

AWS API signatures ensure secure communication between clients and AWS services. Clients include a cryptographic signature in requests, generated from their AWS secret key and the request data. AWS validates this signature with the corresponding AWS access key. This process prevents tampering and unauthorized access to AWS resources. So if you want to make an API request using for example Postman, you have to use their implementation of signature.

Design idea behind Gecholog.ai

Gecholog.ai is designed to be able to use existing sdk or libraries and not introduce a new one to the development process. This could help you implement Gecholog.ai LLM Gateway to manage and monitor your LLM traffic without making radical changes to your existing codes to connect to AWS Bedrock Runtime.

AWS Bedrock Runtime example

We will present two methods to make API request using AWS Python SDK (boto3):

1. Pre-sign request

2. Using invoke_model

The first approach is using the native AWS Python SDK for signing the request only, and a more generic code to make the actual request. This can be beneficial if you want to harmonize how the request is made for example if you decouple part of the code from the LLM API provider or as we will show in this example, if you are using an LLM Gateway like Gecholog.ai and want to be able to use more of the LLM gateway features.

The second approach is using the AWS Python SDK for the entire request, by calling the invoke_model method. Behind the scenes the AWS Python SDK will perform the same steps, first signing the request and then sending it. This approach is recommended by AWS since it takes care of more steps for you in the background, however it also makes your code tighter coupled to AWS as a service provider and as we will show will limit some of the possibilities by using an LLM Gateway such as Gecholog.ai

Pre-sign request

Using AWS Python SDK to pre-sign the request allows you to use generic code to make requests to AWS Bedrock Runtime via Gecholog.ai. Below you have the Python code in detail.

First, import the following packages:

import boto3

from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
from botocore.session import Session
from botocore.credentials import Credentials

import requests
import json

Initialize session with your AWS profile:

profile = 'Your-Profile-Name' 
# Initialize session using your AWS profile
session = boto3.Session(profile_name=profile)

Create variables for the base URL of AWS Bedrock, the API, and the Gecholog URL. In case of local implementation with default configurations, the Gecholog URL should be http://localhost:5380/service/standard/.

bedrock_url = "https://bedrock-runtime.your-region.amazonaws.com/"
api = "model/amazon.titan-text-express-v1/invoke"
gecholog_url = "http://localhost:5380/service/standard/"

No Gecholog specific setup is needed for the request body. Here is an example:

# Create a request body
body = json.dumps({
    "inputText": "Human: explain black holes with 10 words\n\nAssistant:",
    "textGenerationConfig": {
        "maxTokenCount":15,
        "stopSequences":[],
        "temperature":0,
        "topP":0.9,
        },
})

Now you need to create an AWS request object, extract the credentials and the region name from the AWS session you have initialized before, and sign the request by the SigV4Auth method. Remember to add the Content-Type header although it is optional for AWS, as it is required by Gecholog.ai.

# Create an AWS request object
request = AWSRequest(
    method="POST",
    url=bedrock_url + api,
    data=body,
    headers={
        "Content-Type": "application/json", # Required by gecholog
        "Host": bedrock_url.replace("https://", "").rstrip("/")
    }
)


# Get credentials and region from session
credentials = session.get_credentials()
region = session.region_name

# Sign the request
SigV4Auth(credentials, 'bedrock', region).add_auth(request)

Finally, send the pre-signed request via the requests library pre-installed in Python.

# Send the request using requests library
response = requests.post(gecholog_url+api, headers=dict(request.headers), data=request.data)

# Output the response
print(response.text)

You should receive the following json as the response:

{'inputTextTokenCount': 14, 'results': [{'tokenCount': 15, 'outputText': 'Black holes are regions of spacetime with such strong gravity that nothing can escape', 'completionReason': 'LENGTH'}]}

Making pre-signed requests to AWS Bedrock Runtime has the following advantages and disadvantages:

Pro

• Using AWS native libraries

• More agnostic integration code

• Can use both HTTP and HTTPS

• Take advantage of routing capabilities of Gecholog.ai

Con

• Does not use Bedrock methods

• Multi-step process

Invoke model method

You can also create a Bedrock Runtime client by AWS Python SDK and use the invoke_model method (recommneded by AWS) to make requests via Gecholog.ai as proxy, with limited functionalities. In order to integrate Gecholog.ai in this process, you need to first spin up a pre-configured Gecholog.ai container by pulling the source code from Gecholog.ai resources on Github and following chapter 1.a in this guide. Then you need to add a proxy definition to the configurations when you create the client in Python:

import boto3
import json
from botocore.config import Config

proxy_definitions = {
    'https': 'https://localhost:5380', # An empty / router needs to be defined in gl_config.json for gecholog
}

my_config = Config(
    proxies=proxy_definitions,
    proxies_config={
        'proxy_use_forwarding_for_https': True
    }
)

# Trust the self-signed certificate for the proxy for dev purposes
# This is not recommended for production
# You can add certificate information to the Config object to trust the certificate for production
gecholog_client = boto3.client('bedrock-runtime',config=my_config,verify=False)

Now you can start a session, set up request body and make a request through invoke_model method as usual. Again, remember to add the Content-Type header which is required by Gecholog.ai.

# Start an AWS session
boto3.setup_default_session(profile_name='Your-Profile-Name') # Your AWS account profile

body = json.dumps({
    "inputText": "Human: explain black holes with 10 words\n\nAssistant:",
    "textGenerationConfig": {
        "maxTokenCount":15,
        "stopSequences":[],
        "temperature":0,
        "topP":0.9,
        },
})

modelId = 'amazon.titan-text-express-v1'
accept = 'application/json'
contentType = 'application/json' # The contentType header is required by gecholog

# Make the request
response = gecholog_client.invoke_model(body=body,modelId=modelId, accept=accept, contentType=contentType)

response_body = json.loads(response.get('body').read())

# output
print(response_body)

Using the invoke_model method with Gecholog.ai as proxy has both advantages and disadvantages:

Pro

• Simpler integration code

• Closer coupling with the AWS Bedrock method library

• Always HTTPS

Con

• Requires an empty Gecholog.ai router

• Cannot use other routers which can limit Traffic Management capabilities

Conclusion

Gecholog.ai provides two ways to make AWS Bedrock Runtime traffic pass through the LLM Gateway using either pre-signed API request or the invoke_model method via an empty router. Both approaches have advantages and disadvantages, so you can choose the solution that you are more comfortable with.