Unable to run my app or the cookiecutter... getting error `materialize': Could not find

[lanks]

Hi

I have been trying to launch my own app on to lambda and also the cookiecutter but after deploying to AWS I am getting the same error -

/usr/local/lib/ruby/3.2.0/bundler/definition.rb:524:in materialize': Could not find debug-1.8.0, webrick-1.8.1, web-console-4.2.0, capybara-3.39.1, selenium-webdriver-4.9.1, webdrivers-5.2.0, irb-1.6.4, reline-0.3.4, bindex-0.8.1, addressable-2.8.4, matrix-0.4.2, nokogiri-1.15.2-aarch64-linux, regexp_parser-2.8.0, xpath-3.2.0, rexml-3.2.5, rubyzip-2.3.2, websocket-1.2.9, nio4r-2.5.9, websocket-driver-0.7.5, public_suffix-5.0.1 in locally installed gems (Bundler::GemNotFound)`

For some reason it seems like it cannot find the gems.

Does anyone know the fix / why this is happening?

Cheers
Liam

[metaskills]

Morning Liam, a few things to check in your container build.

1. Have you bundled the gems in a local vendor/bundle directory vs system installed?
2. Are you building or copying that entire directory into the container along with the app?
3. Do you have the right BUNDLE_ environment variables and/or local configs to support those paths?

[benbonnet]

@metaskills just getting into lamby, super interested; but getting the same issue : everything deploys to aws, but reading the logs, getting /usr/local/lib/ruby/3.2.0/bundler/definition.rb:524:in 'materialize': Could not find rails-7.0.8, imp... etc

Following the quick start guide, i dont see any specifications about vendoring gems or setting any variables. Is it implied by the fact github workspace is somehow a requirement (at least in the 'quick start' context) ?

[metaskills]

Yup, cover that a bit in this section.

https://lamby.cloud/docs/anatomy#deployment--cicd

The cookie cutter has this and lots more details. See deploy script here.

https://github.com/rails-lambda/lamby-cookiecutter/blob/master/%7B%7Bcookiecutter.project_name%7D%7D/bin/deploy#L26-L31

[benbonnet]

missed those details; will dive into it
thanks alot for your reply!

[benbonnet]

i might lack of understanding, but if i may ask;

Im on my osx laptop (silicon) and not in dev containers. Reading the script (and following the quick start, so running it via ./bin/deploy), it is indeed vendoring the gem etc... so things should be working the same way ? I just dont understand what is missing while not doing it in dev containers.

I had no issues when deploying via devcontainers

[metaskills]

Always fine to ask, happy to help.

I think the issue is "platform" not "arch". Your deployment target "platform" is going to be Linux. If your project has any gems with platform dependencies then you may be compiling gems that are executable and linking on your Mac, not Linux. So let's say you have the MySQL2 gem? And when you vendor that on your Mac it will only run on your Mac and likely links shared objects (system dependences) only present on your Mac. This is why a dev container in both CI/CD is so helpful. Does that help?

[benbonnet]

ok I so got it, stupid reason (it was failing everywhere, git actions, circleci, devcontainers).

The default (mysql2) deploys super fine anywhere.

I use postgres; so I had to extend the Dockerfile.

# Shared image, envs, packages for both devcontainer & prod.
FROM ruby:3.2-bullseye
RUN apt-get update
RUN apt-get install -y postgresql-client
RUN gem install 'aws_lambda_ric'
...

That is, the failing step was RUN gem install 'aws_lambda_ric', with the following :

Error: The command '/bin/sh -c gem install 'aws_lambda_ric'' returned a non-zero code: 1

I kind of get the idea why it was stopping at that step without further infos (the bundle install occurs before the image get effectively used); but that did not helped to spot the (now obvious) issue

I dont have as much time i'd want to dedicate to it; but am hyped, it works. Getting into lambdakiq now

[metaskills]

I'd have to see more output. Seems just fine if I test what is happening during a docker build.

> docker run --interactive --tty --rm --entrypoint "/bin/bash" ruby:3.2-bullseye
root@03998ca6a750:/# apt-get update 1> /dev/null 2> /dev/null
root@03998ca6a750:/# apt-get install -y postgresql-client 1> /dev/null 2> /dev/null
root@03998ca6a750:/# gem install 'aws_lambda_ric'
Fetching aws_lambda_ric-2.0.0.gem
Successfully installed aws_lambda_ric-2.0.0
1 gem installed

A new release of RubyGems is available: 3.4.10 → 3.4.19!
Run `gem update --system 3.4.19` to update your installation.

[jaamqkin]

Hey there! I'm having the same problem as @benbonnet, could you guys help me?
I'm trying to deploy the cookiecutter through Github Actions

This is the output of the error:

== SAM build ==

SAM CLI now collects telemetry to better understand customer needs.

You can OPT OUT and disable telemetry collection by setting the
environment variable SAM_CLI_TELEMETRY=0 in your shell.
Thanks for your help!

Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html

2023-09-26 13:32:04,573 | No config file found in this directory.

2023-09-26 13:32:04,574 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/workspaces/cookiecutter/samconfig.toml'

2023-09-26 13:32:04,575 | Config file location: /workspaces/cookiecutter/samconfig.toml

2023-09-26 13:32:04,575 | Config file '/workspaces/cookiecutter/samconfig.toml' does not exist

2023-09-26 13:32:04,582 | Using SAM Template at /workspaces/cookiecutter/template.yaml

2023-09-26 13:32:04,602 | Using config file: samconfig.toml, config environment: default
2023-09-26 13:32:04,602 | Expand command line arguments to:

2023-09-26 13:32:04,602 | --template_file=/workspaces/cookiecutter/template.yaml --parameter_overrides={'RailsEnv': 'production'} --mount_with=READ --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache

2023-09-26 13:32:04,658 | 'build' command is called

2023-09-26 13:32:04,661 | Collected default values for parameters: {'RailsEnv': 'staging'}

2023-09-26 13:32:04,680 | There is no customer defined id or cdk path defined for resource RailsLambda, so we will use the resource logical id as the resource id
2023-09-26 13:32:04,681 | Unable to resolve property Value: {'Fn::GetAtt': ['RailsLambdaUrl', 'FunctionUrl']}. Leaving as is.
2023-09-26 13:32:04,681 | 0 stacks found in the template
2023-09-26 13:32:04,681 | Collected default values for parameters: {'RailsEnv': 'staging'}

2023-09-26 13:32:04,694 | There is no customer defined id or cdk path defined for resource RailsLambda, so we will use the resource logical id as the resource id
2023-09-26 13:32:04,695 | Unable to resolve property Value: {'Fn::GetAtt': ['RailsLambdaUrl', 'FunctionUrl']}. Leaving as is.
2023-09-26 13:32:04,695 | 1 resources found in the stack
2023-09-26 13:32:04,695 | Found Serverless function with name='RailsLambda' and ImageUri='None'
2023-09-26 13:32:04,695 | --base-dir is not presented, adjusting uri . relative to /workspaces/cookiecutter/template.yaml
2023-09-26 13:32:04,695 | --base-dir is not presented, adjusting uri . relative to /workspaces/cookiecutter/template.yaml

2023-09-26 13:32:04,699 | 1 resources found in the stack
2023-09-26 13:32:04,699 | Found Serverless function with name='RailsLambda' and ImageUri='None'

2023-09-26 13:32:04,699 | Error occurred while trying to track an event: Event 'BuildFunctionRuntime' does not accept value 'None'.
2023-09-26 13:32:04,700 | Instantiating build definitions
2023-09-26 13:32:04,700 | No previous build graph found, generating new one

2023-09-26 13:32:04,700 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(None, /workspaces/cookiecutter, Image, , 4019b7c6-5d0f-482c-9501-238a495eecfb, {'DockerContext': '/workspaces/cookiecutter', 'Dockerfile': 'Dockerfile', 'DockerTag': 'web'}, {}, arm64, []), Function: Function(function_id='RailsLambda', name='RailsLambda', functionname='RailsLambda', runtime=None, memory=1792, timeout=30, handler=None, imageuri=None, packagetype='Image', imageconfig=None, codeuri='/workspaces/cookiecutter', environment={'Variables': {'RAILS_ENV': 'production'}}, rolearn=None, layers=[], events=None, metadata={'DockerContext': '/workspaces/cookiecutter', 'Dockerfile': 'Dockerfile', 'DockerTag': 'web', 'SamResourceId': 'RailsLambda'}, inlinecode=None, codesign_config_arn=None, architectures=['arm64'], function_url_config={'AuthType': 'NONE'}, function_build_info=<FunctionBuildInfo.BuildableImage: ('BuildableImage', 'Regular IMAGE function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None))

2023-09-26 13:32:04,701 | Building codeuri: /workspaces/cookiecutter runtime: None metadata: {'DockerContext': '/workspaces/cookiecutter', 'Dockerfile': 'Dockerfile', 'DockerTag': 'web'} architecture: arm64 functions: RailsLambda
2023-09-26 13:32:04,701 | Building to following folder /workspaces/cookiecutter/.aws-sam/build/RailsLambda
2023-09-26 13:32:04,701 | Building image for RailsLambda function

2023-09-26 13:32:04,703 | Setting DockerBuildArgs: {} for RailsLambda function

2023-09-26 13:32:22,959 | Failed building function RailsLambda
2023-09-26 13:32:22,959 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics

Build Failed

2023-09-26 13:32:22,975 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-09-26 13:32:22,975 | Unable to find Click Context for getting session_id.

2023-09-26 13:32:22,976 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '67333f5a-ac16-4edf-856d-75aadf764361', 'installationId': '55cbc077-4453-4aae-b4ec-f61cbe0b1604', 'sessionId': 'cf091346-c0e8-4c02-a8d8-d900328ff012', 'executionEnvironment': 'Other', 'ci': True, 'pyversion': '3.9.2', 'samcliVersion': '1.97.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '30a7af1f221523c4bb41b4cc6f7dd5f7eff1adf6686ad91d7386f1c98cd78c55', 'initialCommit': None}, 'duration': 18357, 'exitReason': 'DockerBuildFailed', 'exitCode': 1}}]}

2023-09-26 13:32:22,979 | Sending Telemetry: {'metrics': [{'events': {'requestId': '2b9f37ef-5731-4661-ac71-95d7f71e2bb8', 'installationId': '55cbc077-4453-4aae-b4ec-f61cbe0b1604', 'sessionId': 'cf091346-c0e8-4c02-a8d8-d900328ff012', 'executionEnvironment': 'Other', 'ci': True, 'pyversion': '3.9.2', 'samcliVersion': '1.97.0', 'commandName': 'sam build', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '9fd64fccc69244748917aafb5b731f88', 'time_stamp': '2023-09-26 13:32:04.575', 'exception_name': None}]}}}]}

2023-09-26 13:32:23,320 | Telemetry response: 200

Error: The command '/bin/sh -c gem install 'aws_lambda_ric'' returned a non-zero code: 1

2023-09-26 13:32:23,344 | Telemetry response: 200

Error: Dev container exec failed: (exit code: 1)
Error: Dev container exec failed: (exit code: 1)

[hungnv-sr]

Same problem with @lanks.
I also used cookiecutter

The command in bin/deploy looks correct, but I can't run the image because:

/usr/local/lib/ruby/3.2.0/bundler/definition.rb:524:in `materialize': Could not find mysql2-0.5.5, debug-1.8.0, bindex-0.8.1, nio4r-2.5.9, websocket-driver-0.7.6, bigdecimal-3.1.4, psych-5.1.1.1, stringio-3.0.8 in locally installed gems (Bundler::GemNotFound)

Deploying the image to Lambda threw the same error.

[mulvaney]

I had a very similar error when I used the cookiecutter. Without any changes at all, it fails after deployment because gems are missing.

/usr/local/lib/ruby/3.2.0/bundler/definition.rb:524:in materialize': Could not find debug-1.9.0, web-console-4.2.1, capybara-3.39.2, selenium-webdriver-4.10.0, webdrivers-5.3.1, bindex-0.8.1, addressable-2.8.6, matrix-0.4.2, regexp_parser-2.8.3, xpath-3.2.0, rexml-3.2.6, rubyzip-2.3.2, websocket-1.2.10, public_suffix-5.0.4 in locally installed gems (Bundler::GemNotFound)
`

In this case those gems are all in the development/test section of the Gemfile. My lambda has RAILS_ENV=production, so i'm not sure why this is causing problems. I fixed it by commenting out this line in the bin/setup script:

#bundle config --local without 'development test'

When i bundle all the gems (even the dev/test ones) then it gets pass this error.

Doesn't seem to be the same as @hungnv-sr exactly, since that error shows prod gems like mysql2...

[albertski]

I'm also having an issue getting my app to work in Lambda. I see the following error in Cloudwatch:

/usr/local/lib/ruby/3.2.0/bundler/definition.rb:524:in `materialize': Could not find web-console-4.2.1, capybara-3.39.2, selenium-webdriver-4.10.0, webdrivers-5.3.1, bindex-0.8.1, addressable-2.8.5, matrix-0.4.2, regexp_parser-2.8.1, xpath-3.2.0, rexml-3.2.6, rubyzip-2.3.2, websocket-1.2.10, nio4r-2.5.9, websocket-driver-0.7.6, bigdecimal-3.1.4, public_suffix-5.0.3, psych-5.1.0, stringio-3.0.8 in locally installed gems (Bundler::GemNotFound)

The PLATFORMS section of Gemfile.lock:

PLATFORMS
  ruby

[albertski]

I was able to get past this issue (although ran into another issue unrelated) by bundling in a docker container because we need the gems bundled for Linux.

Created a new Dockerfile and called it Dockerfile.bundle

FROM ruby:3.2-bullseye

# Set working directory
WORKDIR /var/task

# Copy Gemfile and Gemfile.lock
COPY Gemfile Gemfile.lock ./

# Install dependencies
RUN bundle install --path vendor/bundle

Updated deploy script with:

echo '== Bundle For Deployment =='
docker build -t bundle-container -f Dockerfile.bundle .
docker run --rm -v $(pwd):/var/task bundle-container sh -c "\
bundle config --global silence_root_warning true && \
bundle config --local deployment true && \
bundle config --local path './vendor/bundle' && \
bundle install --quiet --jobs 4"

[benbonnet]

@albertski can say it works

@metaskills as it is quite hard (impossible?) to do a ./bin/deploy on an M1, that could be interesting to integrate this approach