You are reading the article A Guide To Amazon Indexing & How To Check Your Keyword Indexing updated in November 2023 on the website Cancandonuts.com. We hope that the information we have shared is helpful to you. If you find the content interesting and meaningful, please share it with your friends and continue to follow and support us for the latest updates. Suggested December 2023 A Guide To Amazon Indexing & How To Check Your Keyword Indexing
Amazon customers primarily find products through search. If your product is not indexed for important keywords, you will miss out on potential sales.
Indexing on Amazon is completely different than indexing for Google.
This article will first make sure you have the basics on understanding indexing, then share best practices for getting indexed and checking keyword indexing for an Amazon product detail page.
What Is Indexing on Amazon?With Amazon, all of the data is structured exactly how Amazon wants it to be. This means that indexing on Amazon is not about crawlability, but about making getting the data into the indexing sections of the product detail page.
The length of many of these fields is limited to a specific number of characters. This means the challenge of indexing on Amazon is to make sure that you are maximizing the most important words for potential customers to find your products.
Being indexed for relevant keywords is key to your success.
If your listing is indexed for a keyword, it will show when a customer uses that search term in the Amazon search bar.
In other words, checking your keyword indexing allows you to verify which words your product will be shown as a result for a specific keyword.
As we start to learn about indexing, we first must clarify some common terms:
Keyword: A word or phrase of words used to target potential traffic.
Search Term: The exact word or phrase of words a customer enters into the Amazon search bar.
Learn the Amazon Indexing BasicsIn general, the way customers find products on Amazon is through search.
Think of each search term a customer might use as a door to find your product. The more keywords you are indexed for, the more “doors” a customer has to find you.
Similarly, the more search terms, the more your product will be a result for customer inquiries.
When you are indexed for highly relevant keywords, it will positively impact the amount of traffic and market share you will get on Amazon.
When working on relevancy, you want to prioritize the most relevant words, not simply look at estimated search volume for that keyword or search term. Relevant customers bring in customers more likely to buy.
There is a direct correlation between being indexed for highly relevant purchase intent keywords and your revenue.
For example, let’s say you are selling a baking dish. If you are indexed for the keyword [baking dish], you will show up as a result when a customer enters [baking dish] into the Amazon search bar.
However, if you are only indexed for [baking dish] and not [casserole dish], your product will not show up as a result when a customer enters [casserole dish] into the Amazon search bar.
When a buyer is typing a word or phrase in Amazon for a product, they will get the results based on Amazon’s indexing.
Based on several factors, including past conversion history Amazon will determine which orders will rank at the top of the page.
If your product is not indexed with the right words, buyers will find a competitor’s products or will not find any products related to their search.
Without being indexed for as many highly relevant, purchase intent keywords as possible you will miss out on all the potential traffic through that keyword.
Building up the number of keywords you are indexed for increases the ways your customers find your products.
Opportunities for Indexing on AmazonOn Amazon, you have a limited number of characters or opportunities for keyword indexing.
Amazon limits the length of your titles, bullets, and back-end search terms. You need to focus on keywords that are highly relevant to your product and have search volume.
Amazon rewards conversions and sales, so indexing keywords that drive traffic and do not result in conversions will actually hurt your organic ranking.
Traffic that does not result in conversions will make your products less attractive to the organic search engine on Amazon.
Keyword indexing on Amazon is not a process that you can set and then forget. You want to go back to your keywords and check them regularly for relevancy and conversions.
Some common reasons we might need to add additional keywords into a listing after it was originally optimized are:
We became aware of new alternate uses for a product.
Even after you have created your listings, you want to go back in and check the words your listing is indexing for regularly.
The primary reason to check your keywords regularly is the potential for Amazon to deindex words you were previously indexed for on the website.
Amazon may change its algorithm or make changes to the sub-category structure data. Your data may be deleted when changes in categories or sub-categories occur on the website.
For example, if you sell paper straws, you might know you need to focus on indexing for [paper straw], [biodegradable straw], and [turtle friendly straw]. However, after talking with your customers, you might find that a large number of people are using your paper straws to make pinwheels.
You would then want to adjust your listing to add [pinwheel stick] and [wedding pinwheel supplies] to words you are indexed for on Amazon.
What Does Being Indexed on Amazon Mean?With Amazon, you don’t have to worry about schema mark-up, crawl errors, or canonical issues.
Because Amazon asks for you to upload your data in a specific format, the focus shifts from technical SEO to maximizing the structured data you are giving to Amazon so that you can appear as a result for appropriate search terms.
A finite number of words are used for indexing on your listing. You want to make sure that you are using focused keywords to attract potential buyers.
Being indexed means that typing the keyword into the search bar on Amazon will bring up your product somewhere the search results for that query.
For example, if you index for fuzzy slippers, your product will be in the search results when a buyer looks for fuzzy slippers on Amazon.
You could be position 1 or position 257, but your product is a result for that search term.
Keyword IndexingYou are either indexed for a keyword, or you are not indexed for a keyword. If you are indexed, then your product is in the results; however, if you are not indexed, then your product is not in the search results.
Indexing does not take into account where you appear in the search results.
Keyword RankingThis refers to where your product appears in the search results. Tracking for a keyword will help you determine where your product comes up in the keyword rankings on Amazon.
Your goal when looking at keyword ranking is to improve your ranking for the word or phrases that are relevant to your product.
How Do You Get Indexed on Amazon?The primary method of getting indexed for a keyword is by including the keyword in a data structure. The data structures where you want to put your keyword include:
Your title.
The bullets.
Back-end search terms.
Remaining structured data.
Your title and bullets are where you want to put your most important keywords. Repeating keywords is not a ranking factor on Amazon, so only repeat keywords if it is needed for the customer.
The remaining structured data are fields like color, fabric type, etc. available in the additional sections of the product detail page.
There is a Search Terms field in the flat file. This is a great place to put keywords that didn’t fit in your title and bullets or is for an alternative use case that you may not want to highlight on your product detail page.
Amazon has said in the past that A+ (or Enhanced Brand Content) is not indexed. However, that could always change.
Uploading via Flat Files will allow you to quickly see all of your options for additional keywords like fabric type, genre, sizes, etc.
If you have an existing listing, you can also download the Category Listing Report and Amazon will send you the file for your listing.
Although the primary method of getting indexed relates to the data on your listing, anecdotally, we’ve seen that you can get indexed for words based on customer reviews, questions, and cross-product traffic.
Amazon has not confirmed that customer responses or cross-product traffic relates to keyword indexing. This occurs organically, and it needs to happen over time.
Subcategories impact your keyword relevancy by adding valid value to information or keywords in your structured data.
For example, a grocery subcategory template might have “Salt-Free” as a valid value, where an apparel sub-category template might have fabric type.
Ensuring you are in the correct subcategories allows you to be indexed for additional relevant keywords.
Attempts to bypass a customer focus will put you in a negative situation with Amazon. Do not keyword stuff your listings; focus on appealing to customers with relevant data instead.
Advanced Tip for Indexing with Parent/Child VariationsParent-child variations also impact your indexing. A common misconception about a parent-child variation is that the child will naturally inherit the keywords of the parent product.
That is not always the case.
When you have a parent-child listing with the same product in different colors, you need to check each child product for relevant keywords.
Do not assume that a single child variation with proper indexing will automatically apply to all child variations on your listing.
For example, if you have socks in red, blue, and green and the red sock is indexing for [compression sock], that doesn’t necessarily mean that the blue and green sock will also be indexed for [compression sock].
Each flavor, color, size variation needs to be checked individually for relevant keywords.
Checking for Indexing on Your Amazon ListingOnce you have made changes to your Amazon listing, we recommend that you wait 24 hours before checking to see if you are indexed for the appropriate keywords. You can check keyword indexing for free manually with the following method:
Open an Amazon page and make sure All Categories is selected.
Type in the Amazon Standard Identification Number (ASIN) of the product and keyword you want to check.
Hit Enter.
If the product is indexed for that keyword, it will show us as a result (see image above).
If the product is not indexed for that keyword, the product will not appear in the result.
Suppose you are looking to check indexing for many ASINs or keywords; we have found a paid tool that works well. KW Indexer is a Chrome extension allowing you to type in the keyword and check through the extension.
Watching for Changes in IndexingThere are now tools that can help you verify how your product is indexing for specific keywords.
Helium 10 has a tool called Index Checker that will allow you to upload a list of words and check to see if your listing is indexed for that specific set of words, for example.
You can also use any tool with a Reverse ASIN Search (where the tool looks and finds all of the keyword phrases a listing is indexing for). However, it is important to note that sometimes the data on these tools can be days or sometimes weeks old.
Watch for changes in your traffic or sales. When it comes to monitoring traffic and conversions, we recommend that you watch the Detailed Traffic and Sales Report by Child Product.
This report will show you pages, sessions, orders, and unit session percentages for each child ASIN. (Unit Session Percentage is where Amazon takes the number of orders and sessions you have for an ASIN to determine conversion rates.)
If the number of sessions changes significantly, it may suggest that your product was de-indexed for a primary keyword or an algorithm change that resulted in a different page ranking.
It may also relate to a merging of two separate sub-categories.
Ensuring that you are appropriately indexed is one of the most accessible solutions to make your product easy to find.
More Resources:
Image Credits
All screenshots taken by author, January 2023 and April 2023
You're reading A Guide To Amazon Indexing & How To Check Your Keyword Indexing
The Ultimate Guide To Creating A True ‘Ultimate Guide’
From choosing the right keywords, to conducting audience research, to writing engaging titles, it can be a long, painstaking process to create content that outranks your top-ranking competitors.
This desire to create bigger and better content has led to this often cringe-worthy trend: turning every article into an “ultimate guide”.
Is There Such a Thing as an ‘Ultimate’ Guide?“Ultimate guide” implies that your content is the best, right? That no one else can top it. That it’s the blog post to end all blog posts.
The issue is that with so many articles claiming to be “ultimate guides”, throwing this into your post title has started to lose its meaning. Claiming that your article is the best doesn’t make it so.
It’s in the value.
Here’s how to create a true ultimate guide – the right way – in order to attract more organic traffic and get readers raving about your content.
An Ultimate Guide Is More Than Just a ListicleMany content creators and SEO professionals approach content with the belief that longer is better – despite the numerous studies that have shown diminishing returns on content that is too long.
There’s a sweet spot for every topic and niche, and simply adding more bullet points for “top tips” to your ultimate guide does not necessarily mean you are adding more value.
Your goal in creating an ultimate guide is to provide better information than all of the top-ranking posts for your chosen focus keyword.
You do that not by making it longer (necessarily), but by doing your research to determine what the existing articles offer that your guide has to have.
Step 1: Have a Purpose (Beyond Ranking)I get it. It’s enticing to go after those sweet money keywords when you see an opportunity to outrank your competitors.
But beyond rankings, there’s typically the ultimate goal of getting readers to take action – by subscribing, by buying a project, or by filling out a form.
That’s why it’s short-sighted to create an ultimate guide for the sole purpose of targeting a particular keyword. Instead, it’s best to have a strategy that works to drive conversions through your content.
Before writing your ultimate guide, consider what action (if any) you want readers to take. Are you:
Simply aiming to inform readers about a topic?
Trying to build brand trust?
It’s essential that you know your end goal.
Having a clear objective in mind will help you determine what value you need to provide and how. It will keep you focused on writing content that serves to not only attract traffic but also make the most of that traffic.
Step 2: Choose Your Keyword(s)Not every keyword is suited for an ultimate guide.
Just because your goal is to provide more value than the content that’s already ranking, doesn’t mean throwing “ultimate guide” in front of it is the move. It may be better suited for a different type of content (like a standard blog post or a listicle).
“Ultimate guide”-type keywords are ones that warrant a well-thought-out, in-depth piece of content. They should be topics that necessarily require a comprehensive guide in order to leave readers feeling sufficiently informed.
Relatively broad topics are a great fit an ultimate guide because they require many sections, explanations, examples, and actionable steps.
“Best places to visit in Thailand” can be summed up in a relatively concise listicle. “How to travel in Thailand” leaves much more room for elaboration.
Consider whether an ultimate guide would do your chosen keyword justice, or whether a different type of content would be a better fit instead. Some things to consider are:
What is the user intent behind the keyword?
Could you answer the user’s primary question in a relatively short (or standard-sized) article?
Can you effectively write a comprehensive guide about the topic without stuffing it full of fluff?
Are there already tons of “ultimate guides” about this topic?
Are you confident that you can at least offer 2X the value of the top-ranking articles?
In summary, choose keywords that warrant a comprehensive piece of content.
If you feel that in order to create an ultimate guide you’ll have to chock your content full of useless facts and tips, you should probably take a different approach.
Step 3: Scope out the Not-So-Ultimate GuidesNot all ultimate guides are created equal.
If you’ve decided that your focus keyword warrants an ultimate guide, then your next step should be to look at what’s already ranking. Chances are, you’ll find some existing “ultimate guides”.
In this step, you’ll want to look at what your competitors are doing right and wrong.
Are they simply regurgitating the same-old information?
Are they bringing anything unique to the table?
How can you learn from them and make your content 2X better?
Since your competitors are ranking for that keyword, they must be doing something right.
Look for similarities and characteristics that you think may be giving them a competitive edge. Then, looks for gaps in information that you can fill with your ultimate guide.
Your ultimate guide – as the name implies – should aim to be the best of the best. That means capitalizing off of what your competitors are doing right and then making your content even better.
Step 4: Survey Your AudienceWant to know what your audience really wants to read? Ask them!
The best way to provide the most value in your ultimate guide is to ask your readers what they are interested in.
You can survey your audience casually (by asking them questions on social media) or by sending them an official survey (directly or via email). Their answers will help you determine what information to include in your guide.
Here are some questions you can ask your audience:
What is your top struggle when it comes to dealing with [ topics/problem ]?
What are three questions you have about [ topic ]?
What do you wish other brands knew about [ topic ]?
What are some solutions you have tried to solve [ problem ]?
What’s your top tip when it comes to [ topic ]?
In SEO content, it’s best to avoid making guesses when you can get your answers straight from the source.
By asking your audience directly, you can be sure to write content that answers their most burning questions, thoroughly covers the topic, and provides the value that your particular audience is looking for.
Step 5: Include Case Studies, Examples & StatisticsAny claims that you make in your ultimate guide should be supported by case studies, examples, or statistics.
Including this information serves to provide value to your readers in a variety of ways:
AuthorityIncluding sources shows your readers that you know what you are talking about.
In other words, it helps establish you as an authority on the topic.
Rather than making baseless claims, you can back up all of your information with facts.
TrustReferencing sources helps build trust with your audience.
When people are confident that the information you are providing is accurate, they are more likely to buy from you.
ApplicationIncluding examples helps readers make the connection between the information in your ultimate guide and how they can apply it to their lives or businesses.
Examples prove that the tips or strategies you are sharing work and that by taking action readers can see similar results.
E-A-T“E-A-T” is a principle in SEO that stands for Expertise, Authoritativeness, and Trustworthiness.
Google’s Danny Sullivan has suggested that the secret to better rankings is better content. And the key to better content? You can find that in the quality raters guidelines, which talks in great detail about E-A-T.
By including case studies, examples, and sources in your ultimate guide, you’ll indicate to search engines that your content is valuable, accurate, and comes from a reputable source.
Following this step is another way to provide value where your competitors may be coming up short.
Step 6: Make It ActionableIf you know the search intent behind the keyword you are targeting with your ultimate guide, then you should be able to make a guess about what users are trying to achieve by reading your content.
Most times, their goal is to get their most pressing questions answered. Why? Because they are likely experiencing an issue that needs fixing, or want to learn more about a topic.
Your best bet in creating your ultimate guide, then, is to make the information you provide actionable in some way. You want to do your very best to help them fix whatever problem they are experiencing and answer every question they may have.
If your ultimate guide is about “how to plan a honeymoon”, you’ll likely want to include tips like:
Where and how to hire a travel planner.
How to choose the best hotel.
How to prepare for international travel.
How to budget for a honeymoon.
Best places to book affordable airfare.
Even better, these actionable steps all provide an opportunity for you to link to other related content on your site. Either way, you are giving the readers the information they need to apply your tips to their lives.
Not only will making it actionable put your guide miles above competing articles, but it’ll make it more likely that your guide will drive conversions.
If you are able to offer all of the information they need on your site, it’ll be easy to direct them to your products and done-for-you services.
Step 7: Make It PrettyCreating an ultimate guide takes time – and, often, money. To truly make an “ultimate” guide, you won’t want to cut corners.
Make the most of the work and resources you are putting in by making your guide attractive and eye-catching, as well as informative. It’ll be worth it to employ a graphic designer and/or web developer to create a sleek, on-brand looking guide.
Including various types of media – like videos, infographics, and images – will also help you repurpose your content across multiple platforms. You can build out an entire content marketing strategy from a single piece of content.
Step 8: Add a Content UpgradeAssuming your content is optimized to the nines and starts generating organic traffic, you’ll also want to be sure you are making the most of that traffic.
Beyond including links to your product pages or adding calls-to-action, you can incorporate a content upgrade to get readers to join your email list.
Content upgrades work great because they entice readers with even more valuable content than what you are already offering in your ultimate guide. If the upgrade is irresistible, users will hand over their email addresses in exchange for more juicy content.
Some possible content upgrade ideas include:
A free downloadable ebook.
A link to an exclusive expert interview.
A free sign-up for a 5-day email course.
A discount code for a paid product.
An expanded ultimate guide (e.g. “Get the rest of the tips here!”).
A free design or strategy template.
By adding a content upgrade to your ultimate guide, you can capture some of that traffic coming in and remarket to them later.
You’re essentially giving users multiple ways to opt-in (convert) – ensuring that you are making the absolute most of the traffic your content is generating.
Examples of Ultimate Guides That Get It RightIf you want to create the ultimate “ultimate guide”, it’s recommended that you learn from those that are already doing it right.
After following the tips above, look to some of these examples to find ways to make your guide even better.
Create Your Ultimate ‘Ultimate Guide’Think you have what it takes to create a true “ultimate guide”?
Remember that your primary objective is to provide even more value than what’s already ranking.
Simply calling your guide an “ultimate guide” isn’t enough.
Follow the steps above in order to target the best keyword, create content your audience truly wants to read and provide accurate, actionable information that grabs users’ (and Google’s) attention.
More Resources:
Image Credits
A Complete Guide To Vmware Benefits
Introduction to VMware
Web development, programming languages, Software testing & others
This software helps us in various domains like security, storage, networking, etc. VMware provides us with various software and products that can be used for different benefits; here, we will see the various benefits of using that product and software for better understanding and usage.
Various VMware BenefitsAs we already know, VMware has many benefits, which can be understood by the various product it provides, which adds great help to security networking, storage, and many more areas.
1. Provides virtual desktop infrastructureOne of the benefits of using this is we can use the desktop from anywhere we want. From this, we do not require a full desktop setup in the workplace; we can use VMware Horizon, which allows us to manage and run the Windows desktop from VMware Cloud or AWS. This removes a lot of things for us, like we do not require to manage and set up the full desktop at the workplace. Also, it helps reduce the monitoring and managing of user security and centralizes management. We can use this with two more VMware products, Dynamic Environment Manager and App Volumes, which help us in application delivery and managing the Windows desktop.
2. Provide personal desktopVMware created this as their first product, enabling users to run or manage virtual machines directly on a single Linux, Windows, or laptop. Using this, we can have a virtual machine inside the physical machine, which can run without causing any issues; in short, it can run parallel or simultaneously. If we talk about virtual machines, they have operating systems such as Linux or Windows. With this, we can even run Windows on the Linux machine and vice versa without worrying about the installed operating system on the machine. The product name VM Workstation enables us to run the virtual machine in the machine; for Mac computers, we have VM Fusion.
3. Provide storage and availability 4. Provide disaster recovery managementVMware benefits also include disaster recovery; for this, it provides us with the Site Recovery Manager, which helps us create the recovery plan, which will be executed automatically in the case of failure. The NSX further integrates with this system to maintain and preserve the security and network on the migrated VMs.
5. Provide the cloud infrastructureFor infrastructure, we have one product from VMware which is known as vSphere, which provide the following points:
vMotion
vSphere Client
ESXi
vCenter Server
6. Provide us SDDC platformSDDC manager helps to integrate various software into a single platform, such as VMware NSX, vSphere, vSAN, etc. So for this, we have VMware cloud foundation, which is a software that helps to bundle this mentioned software by the use of the SDDC platform; now we can deploy this bundle on the private cloud or also have the option to run this bundle within as public cloud but as a service. Admin can do all these tasks; admin also has the provision to the application without the need for storage and network.
7. Provide network and securityAs seen above are the main benefits of VM, as we have already seen it provides us with many products which can be used for different purposes as per the need, one of the main things about doing things virtually without carrying the setup at the workplace.
Below are the key points that need to be kept in mind while using the VN product; they provide us with many benefits, but we also have some drawbacks that must be focused on.
Also, there is a lack of support, which means we may encounter several bugs while using the VM product.
Not all things are free; the fees are very high for licensing.
Conclusion – VMware BenefitsAs we have already seen so many benefits of VM in this article, we have also seen the different products that provide for different purposes; you can understand and start using them by the above explanation; we have many more things in VM.
Recommended ArticlesThis is a guide to VMware Benefits. Here we discuss the introduction and various VMware benefits for better understanding. You may also have a look at the following articles to learn more –
A Practical Guide To Digital Transformation
Pierre Nanterme, CEO of Accenture, once said, ‘Digital (technology) is the main reason just over half of the companies on the Fortune 500 have disappeared since the year 2000’. Does your board have an up-to-date digital transformation strategy and framework? In this guide to digital transformation we look at the key areas board members should focus on and recommend training in this area, specifically designed for board members.
70% of the top global brands are now digital and tech companies. 64% of the people on the planet have access to a smartphone. If your organisation doesn’t have the desire and the strategy to meet the disruptive power of digital, then you are likely to be in trouble.
As a board member, digital transformation must be an always-on board agenda item. Here, we explain why and examine the questions board members should ask regarding digital transformation.
Examples of transformationAirbnb is an excellent example of a startup that disrupted the hotel sector without building any hotels.
On the other hand, General Electric is an example of a massive organisation that faced disruption but battled to keep up. Watch Robert Farrell (below) discuss how this example of digital transformation went badly wrong.
What does digital disruption mean?Digital disruption is ‘an effect that changes the fundamental expectations and behaviours in a culture, market, industry or process caused by, or expressed through, digital capabilities, channels or assets.’ It fundamentally changes your market and industry.
For example, look at what happened to Blockbuster in the U.S. An organisation and its board were so myopic, locked into groupthink and their traditional business model, that they couldn’t see Netflix as a threat. And we know how that ended. Netflix fundamentally disrupted the TV and film rental business, and Blockbuster went bust. Read the entire case study here.
What does digital transformation mean?Digital transformation is “the process of exploiting digital technologies and supporting capabilities to create a robust new business model”.
Digital transformation is your response to changes in your market. It’s not just about technology; it’s how you adapt your organisation, its people and processes. It is a fundamental re-think of your value proposition and business model.
What are the types of digital transformation?
Process transformation.
Business model transformation.
Domain transformation.
Cultural and organisational transformation.
The digital journeyWhat can boards do to ensure their organisations embrace digital transformation?
Your board needs to ensure it has the right resources and plans to deal with digital disruption.
Next, the board needs to ask, how attuned is the management team to the potential disruption of your organisation and industry? As a first step, your board could set up a digital trends committee to monitor and report back on how digital will transform your sector.
Individual board members also need to keep themselves informed about digital trends and their potential impacts on their industry. Don’t rely on briefings from the executive.
A digital transformation frameworkThe board must also ask itself, would it support the management team to undertake potentially disruptive and innovative projects if they could offer long-term benefits? What’s your risk appetite when it comes to innovation? Does your board have access to your organisation’s digital transformation framework? As a board member, you need to be clear about that.
Another question the board should ask is, what are the biggest challenges the organisation faces when adopting emerging technologies? Are the challenges to do with hiring new people, obtaining sufficient capital, and integrating new technologies into older systems?
Digital transformation trends boards should study
1. Cloud technology
2. The hybrid work model
3. Artificial Intelligence and Machine Learning
4. Cybersecurity and privacy
5. Blockchain technology, NFTs And the Metaverse
As a board member, you must ensure that the broad topic of digital transformation is always on the board’s agenda. It also needs to be part of your organisation’s risk management. And finally, the board should recruit non-executive directors with specific technical skills to ensure that the executive asks the right questions when discussing digital disruption and transformation.
Digital transformation training
In today’s world, being able to harness digital transformation and implement it is a must. It impacts your entire organisation, supply chains, value propositions, working environment, and business models.
The Corporate Governance Institute has created a self-paced, fully online certificate in digital transformation in collaboration with digital transformation experts and partners.
Become the digital transformation expert in your organisation. Show the board how it can change the way your business creates value and how to measure success and identify emerging risks.
Be the person who drives this critical strategic issue.
A Comprehensive Guide To Reinforcement Learning
Everyone heard when DeepMind announced its milestone project AlphaGo –
AlphaGo is the first computer program to defeat a professional human Go player, the first to defeat a Go world champion, and is arguably the strongest Go player in history.
This alone says a lot about how powerful the program itself is but how did they achieve it? They did it through novel approaches in Reinforcement learning!
And it’s not just fixated on games, the applications range from –
In this guide, I’ll walk you through the theory behind reinforcement learning, ideas based on theory, various algorithms with basic concepts, and implementation in Python!
Table of Contents
Fundamentals of Reinforcement learning
Creating an environment using OpenAI Gym
Algorithms (Concepts and Implementation)
RL – Libraries in Python
Challenges in Reinforcement Learning
Conclusion
Fundamentals of Reinforcement Learning
Let’s dig into the fundamentals of RL and review them step by step.
Key elements fundamental to RLThere are basically 4 elements – Agent, Environment, State-Action, Reward
Agent
An agent is a program that learns to make decisions. We can say that an agent is a learner in the RL setting. For instance, a badminton player can be considered an agent since the player learns to make the finest shots with timing to win the game. Similarly, a player in FPS games is an agent as he takes the best actions to improve his score on the leaderboard.
Environment
For instance, we discussed badminton players, here the court is the environment in which the player moves and takes appropriate shots. Same in the case of the FPS game, we have a map with all the essentials (guns, other players, ground, buildings) which is our environment to act for an agent.
State – Action
A state is a moment or instance in the environment at any point. Let’s understand it with the help of chess. There are 64 places with 2 sides and different pieces to move. Now this chessboard will be our environment and player, our agent. At some point after the start of the game, pieces will occupy different places in the board, and with every move, the board will differ from its previous situation. This instance of the board is called a state(denoted by s). Any move will change the state to a different one and the act of moving pieces is called action (denoted by a).
Reward
We have seen how taking actions change the state of the environment. For each action ‘a’ the agent takes, it receives a reward (feedback). The reward is simply a numerical value assigned which could be negative or positive with different magnitude.
Let’s take badminton example if the agent takes the shot which results in a positive score we can assign a reward as +10. But if it gets the shuttle inside his court then it will get a negative reward -10. We can further break rewards by giving small positive rewards(+2) for increasing the chances of a positive score and vice versa.
Rough Idea to relate Reinforcement Learning problemsBefore we move on to the Math essentials, I’d like to give a bird-eye view of the reinforcement learning problem. Let’s take the analogy of training a pet to do few tricks. For every successful completion of the trick, we give our pet a treat. If the pet fails to do the same trick we don’t give him a treat. So, our pet will figure out what action caused it to receive a cookie and repeat that action. Thus, our pet will understand that completing a trick caused it to receive a treat and will attempt to repeat doing the tricks. Thus, in this way, our pet will learn a trick successfully while aiming to maximize the treats it can receive.
Here the pet was Agent, groundfloor our environment which includes our pet. Treats given were rewards and every action pet took landed him in a different state than the previous.
Markov Decision Process (MDP)The Markov Decision Process (MDP) provides a mathematical framework for solving RL problems. Almost all RL problems can be modeled as an MDP. MDPs are widely used for solving various optimization problems. But to understand what MDP is, we’d have to understand Markov property and Markov Chain.
The Markov property and Markov chain
Markov Property is simply put – says that future states will not depend on the past and will solely depend on the present state. The sequence of these states (obey Markov property) is called Markov Chain.
Change from one state to another is called transition and the probability of it is transition probability. In simpler words, it means in every state we can have different choices(actions) to choose from. Each choice(action) will result in a different state and the probability of reaching the next state(s’) will be stored in our sequence.
Now, if we add rewards in Markov Chains we get a sequence with the state, transition probability, and rewards (The Markov Reward Process). If we further extend this to include actions it will become The Markov Decision Process. So, MDP is just a sequence of . We will learn more concepts on the go as we move further.
OpenAI Gym for Training Reinforcement Learning AgentsOpenAI is an AI research and deployment company whose goal is to ensure that artificial general intelligence benefits all of humanity. OpenAI provides a toolkit for training RL agents called Gym.
As we have learned that, to create an RL model we need to create an environment first. The gym comes into play here and helps us to create abstract environments to train our agents on it.
Installing Gym
Overview of Gym
Creating an episode in the Gym environment
Cart-Pole balancing with a random agent
Installing Gym
Its installation is simple using Pip. Though the latest version of Gym was just updated a few days ago after years, we can still use the 0.17 version.
pip install gymYou can also clone it from the repository.
Creating our first environment using GymWe will use pre-built (in Gym) examples. One can get explore all the agents from OpenAI gym documentation. Let’s start with Mountain Car.
First, we import Gym
import gymTo create an environment we use the ‘make’ function which required one parameter ID (pre-built ones can be found in the documentation)
env = gym.make('CartPole-v0')To can see how our environment actually looks like using render function.
env.render()The goal here is to balance the pole as long as possible by moving the cart left or right.
To close rendered environment, simply use
env.close() Cartpole-Balancing using Random Agent import gym env = gym.make('CartPole-v0') env.reset() for _ in range(1000): env.render() env.step(env.action_space.sample()) # take a random action env.close()We created an environment, the first thing we do is to reset our environment to its default values. Then we ran it for 1000 timesteps by taking random actions. The ‘step’ function is basically transitioning our current state to the next state by taking the action our agent gives (in this case it was random).
Observations
If we want to do better than just taking random actions, we’d have to understand what our actions are doing to the environment.
The environment’s step function returns what we need in the form of 4 values :
observation (object): an environment-specific object representing the observation of our environment. For example, state of the board in a chess game, pixels as data from cameras or joints torque in robotic arms.
reward (float): the amount of reward achieved by each action taken. It varies from env to env but the end goal is always to maximize our total reward.
done (boolean): if it’s time to reset our environment again. Most of the tasks are divided into a defined episode (completion) and if done is true it means the env has completed the episode. For example, a player wins in chess or we lose all lives in the Mario game.
info (dict): It is simply diagnostic information that is useful for debugging. The agent does not use this for learning, although it can be used for other purposes. If we want to extract some info from each timestep or episode it can be done through this.
This is an implementation of the classic “agent-environment loop”. With each timestep, the agent chooses an action, and the environment returns an observation and a reward with info(not used for training).
The whole process starts by calling the reset() function, which returns an initial observation.
import gym env = gym.make('CartPole-v0') for i_episode in range(20): observation = chúng tôi for t in range(100): env.render() #renders our cartpole env print(observation) action = env.action_space.sample() #takes random action from action space observation, reward, done, info = env.step(action) if done: #prints number of timesteps it took to finish the episode print("Episode finished after {} timesteps".format(t+1)) break env.close()Now, what we see here is observation at each timestep, in Cartpole env observation is a list of 4 continuous values. While our actions are just 0 or 1. To check what is observation space we can simply call this function –
import gym env = gym.make('CartPole-v0') print(env.action_space) #type and size of action space print(env.observation_space) #type and size of observation spaceDiscrete and box are the most common type of spaces in Gym env. Discrete as the name suggests has defined values while box consists of continuous values. Action values are as follows –
Value Action 0 Push cart towards the left 1 Push cart towards the right
Meanwhile, the observation space is a Box(4,) with 4 continuous values denoting –
0.02002610 -0.0227738 0.01257453 0.04411007 Position of Cart Velocity of Cart Angle of Pole The velocity of Pole at the tipGym environments are not just restricted to text or cart poles, its wide range is as follows –
Atari games Box2D MuJoCoAnd many more… We can also create our own custom environment in the gym suiting to our needs.
Popular Algorithms in Reinforcement LearningIn this section, I will cover popular algorithms commonly used in Reinforcement Learning. Right after some basic concepts, it will be followed with implementation in python.
Deep Q NetworkThe objective of reinforcement learning is to find the optimal policy, that is, the policy that gives us the maximum return (the sum of total rewards of the episode). To compute policy we need to first compute the Q function. Once we have the Q function, then we can create a policy that selects the best action based on the maximum Q value. For instance, let’s assume we have two states A and B, we are in state A which has 4 choices, and corresponding to each choice(action) we have a Q value. In order to maximize returns, we follow the policy that has argmax (Q) for that state.
State Action Value A left 25 A Right 35 A up 12 A down 6We are using a neural network to approximate the Q value hence that network is called the Q network, and if we use a deep neural network to approximate the Q value, then it is called a deep Q network or (DQN).
Basic elements we need for understanding DQN is –
Replay Buffer
Loss Function
Target Network
Replay Buffer –
We know that the agent makes a transition from a state s to the next state 𝑠′ by performing some action a, and then receives a reward r. We can save this transition information in a buffer called a replay buffer or experience replay. Later we sample random batches from buffer to train our agent.
We learned that in DQN, our goal is to predict the Q value, which is just a continuous value. Thus, in DQN we basically perform a regression task. We generally use the mean squared error (MSE) as the loss function for the regression task. We can also use different functions to compute the error.
Target Network –
There is one issue with our loss function, we need a target value to compute the losses but when the target is in motion we can no longer get stable values of y_i to compute loss, so here we use the concept of soft update. We create another network that updates slowly as compared to our original network and computes losses since now we have frozen values of y_i. It will be better understood with the code below.
Let’s start coding our DQN algorithm!
import random import gym import numpy as np from collections import deque from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Flatten, Conv2D, MaxPooling2D , Dense, Activation from tensorflow.keras.optimizers import Adam env = gym.make("MsPacman-v0") state_size = (88, 80, 1) #defining state size as image input pixels action_size = env.action_space.n #number of actions to be takenPre-processing to feed image in our CNN
color = np.array([210, 164, 74]).mean() def preprocess_state(state): #creating a function to pre-process raw image from game #cropping the image and resizing it image = state[1:176:2, ::2] #converting the image to greyscale image = image.mean(axis=2) #improving contrast image[image==color] = 0 #normalize image = (image - 128) / 128 - 1 #reshape and returning the image in format of state space image = np.expand_dims(image.reshape(88, 80, 1), axis=0) return imageWe need to pre-process the raw image from the game, like removing color, cropping to the desired area, resizing it to state space as we defined previously.
Building DQN class
#epsilon of 0.8 denotes we get 20% random decision self.epsilon = 0.8 #define the update rate at which we update the target network self.update_rate = 1000 #building our main Neural Network self.main_network = self.build_network() #building our target network (same as our main network) self.target_network = self.build_network() #copying weights to target network self.target_network.set_weights(self.main_network.get_weights()) def build_network(self): #creating a neural net model = Sequential() model.add(Conv2D(32, (8, 8), strides=4, padding='same', input_shape=self.state_size)) model.add(Activation('relu')) #adding hidden layer 1 model.add(Conv2D(64, (4, 4), strides=2, padding='same')) model.add(Activation('relu')) #adding hidden layer 2 model.add(Conv2D(64, (3, 3), strides=1, padding='same')) model.add(Activation('relu')) model.add(Flatten()) #feeding flattened map into our fully connected layer model.add(Dense(512, activation='relu')) model.add(Dense(self.action_size, activation='linear')) #compiling model using MSE loss with adam optimizer return model #we sample random batches of data, to store whole transition in buffer def store_transistion( self, state, action, reward, next_state, done ): self.replay_buffer.append(( state, action, reward, next_state, done)) # defining epsilon greedy function so our agent can tackle exploration vs exploitation issue def epsilon_greedy(self, state): #whenever a random value < epsilon we take random action if random.uniform(0,1) < self.epsilon: return np.random.randint(self.action_size) #then we calculate the Q value Q_values = self.main_network.predict(state) return np.argmax(Q_values[0]) #this is our main training function def train(self, batch_size): #we sample a random batch from our replay buffer to train the agent on past actions minibatch = random.sample(self.replay_buffer, batch_size) #compute Q value using target network for state, action, reward, next_state, done in minibatch: #we calculate total expected rewards from this policy if episode is not terminated if not done: target_Q = (reward + self.gamma * np.amax(self.target_network.predict(next_state))) else: target_Q = reward #we compute the values from our main network and store it in Q_value Q_values = self.main_network.predict(state) #update the target Q value for losses Q_values[0][action] = target_Q #training main network self.main_network.fit(state, Q_values, epochs=1, verbose=0) #update the target network weights by copying from the main network def update_target_network(self): self.target_network.set_weights(self.main_network.get_weights()Now we train our network after defining the values of hyper-params
num_episodes = 500 #number of episodes to train agent on num_timesteps = 20000 #number of timesteps to be taken in each episode (until done) batch_size = 8 #taking batch size as 8 num_screens = 4 #number of past game screens we want to use dqn = DQN(state_size, action_size) #initiating the DQN class done = False #setting done to false (start of episode) time_step = 0 #begining of timestep for i in range(num_episodes): #reset total returns to 0 before starting each episode Return = 0 #preprocess the raw image from game state = preprocess_state(env.reset()) for t in range(num_timesteps): env.render() #render the env time_step += 1 #increase timestep with each loop #updating target network if time_step % dqn.update_rate == 0: dqn.update_target_network() #selection of action based on epsilon-greedy strategy action = dqn.epsilon_greedy(state) #saving the output of env after taking 'action' next_state, reward, done, _ = env.step(action) #Pre-process next state next_state = preprocess_state(next_state) #storing transition to be used later via replay buffer dqn.store_transistion(state, action, reward, next_state, done) #updating current state to next state state = next_state #calculating total reward Return += reward if done: print('Episode: ',i, ',' 'Return', Return) #if episode is completed terminate the loop break #we train if the data in replay buffer is greater than batch_size #for first 1-batch_size we take random actions dqn.train(batch_size)Results – Agent learned to play the game successfully.
DDPG (Deep Deterministic Policy Gradient)DQN works only for discrete action space but it’s not always the case that we need discrete values. What if we want continuous action output? to overcome this situation, we start with DDPG (Timothy P. Lillicrap 2023) to deal with when both state and action space is continuous. The idea of replay buffer, target functions, loss functions will be taken from DQN but with novel techniques which I will explain in this section.
Now, we move on to the core Actor-critic method. The original paper explains this concept quite well, but here is a rough idea. The actor takes a decision based on a policy, critic evaluates state-action pair, and gives it a Q value which is assigned to each pair. If the state-action pair is good enough according to critics, it will have a higher Q value (more preferable) and vice versa.
Critic Network
#creating class for critic network class CriticNetwork(nn.Module): def __init__(self, beta): super(CriticNetwork, self).__init__() #fb, insta as state of 2 dim self.input_dims = 2 #hidden layers with 256 N self.fc1_dims = 256 #hidden layers with 256 N self.fc2_dims = 256 #fb, insta spends as 2 actions to be taken self.n_actions = 2 # state + action as fully connected layer chúng tôi = nn.Linear( 2 + 2, self.fc1_dims ) #adding hidden layers chúng tôi = nn.Linear(self.fc1_dims, self.fc2_dims) #final Q value from network self.q1 = nn.Linear(self.fc2_dims, 1) #using adam optimizer with beta as learning rate self.optimizer = optim.Adam(self.parameters(), lr=beta) #device available to train on CPU/GPU self.device = T.device('cuda' if T.cuda.is_available() else 'cpu') #assigning device self.to(self.device) #Creating Critic Network with state and action as input def CriticNetwork(self, state, action): #concatinating state and action before feeding to Neural Net q1_action_value = self.fc1(T.cat([state, action], dim=1 )) q1_action_value = F.relu(q1_action_value) #adding hidden layer q1_action_value = self.fc2(q1_action_value) q1_action_value = F.relu(q1_action_value) #getting final Q value q1 = self.q1(q1_action_value) return q1Now we move to actor-network, we created a similar network but here are some key points which you must remember while making the actor.
Weight initialization is not necessary but generally, if we provide initialization it tends to learn faster.
Choosing an optimizer is very very important and results can vary from the optimizer to optimizer.
Now, how to choose the last activation function solely depends on what kind of action-space, you are using, for example, if it is small and all values are like [-1,-2,-3] to [1,2,3] you can go ahead and tanh (squashing) function, but if you have [-2,-40,-230] to [2,60,560] you might want to change the activation function or create a wrapper.
Actor-Network
class ActorNetwork(nn.Module): #creating actor Network def __init__(self, alpha): super(ActorNetwork, self).__init__() #fb and insta as 2 input state dim self.input_dims = 2 #first hidden layer dimension self.fc1_dims = fc1_dims #second fully connected layer dimension self.fc2_dims = fc2_dims #total number of actions self.n_actions = 2 #connecting fully connected layers chúng tôi = nn.Linear(self.input_dims, self.fc1_dims) chúng tôi = nn.Linear(self.fc1_dims, self.fc2_dims) #final output as number of action values we need (2) chúng tôi = nn.Linear(self.fc2_dims, self.n_actions) #using adam as optimizer self.optimizer = optim.Adam(self.parameters(), lr=alpha) #setting up device (CPU or GPU) to be used for computation self.device = T.device('cuda' if T.cuda.is_available() else "cpu") self.to(self.device) #connecting the device def forward(self, state): #taking state as input to our fully connected layer prob = self.fc1(state) #adding activation layer prob = F.relu(prob) #adding second layer prob = self.fc2(prob) prob = F.relu(prob) #fixing each output between 0 and 1 mu = T.sigmoid(self.mu(prob)) return muNote: We used 2 hidden layers since our action space was small and our environment was not very complex. Authors of DDPG used 400 and 300 neurons for 2 hidden layers but we can increase at the cost of computation power.
Just like gym env, agent has some conditions too. We initialized our target networks with same weights as our original (A-C) networks. Since we are chasing a moving target, target networks create stability and helps original networks to train.
We initialize all the basic requirements, as you might have noticed we have a loss function parameter too. We can use different loss functions and choose whichever works best (can be L1 smooth loss), paper used mse loss, so we will go ahead and use it as default.
Here we include the ‘choose action’ function, you can create an evaluation function as well to cross-check values that outputs action space without noise.
‘Update parameter’ function, now this is where we do soft (target networks) and hard updates (original networks, complete copy). Here it takes only one parameter Tau, this is similar to how we think of learning rate.
It is used to soft update our target networks and in the paper, they found the best tau to be 0.001 and it usually is the best across different papers.
class Agent(object): #binding everything we did till now def __init__( self, alpha , beta, input_dims= 2, tau, env, gamma=0.99, n_actions=2, max_size=1000000, batch_size=64): #fixing discount rate gamma self.gamma = gamma #for soft updating target network, fix tau chúng tôi = tau #Replay buffer with max number of transitions to store self.memory = ReplayBuffer(max_size) #batch size to take from replay buffer self.batch_size = batch_size #creating actor network using learning rate alpha self.actor = ActorNetwork(alpha) #creating target network with same learning rate self.target_actor = ActorNetwork(alpha) #creating critic network with beta as learning rate self.target_critic = CriticNetwork(beta) #adjusting scale as std for adding noise self.scale = 1.0 self.noise = np.random.normal(scale=self.scale,size=(n_actions)) #hard updating target network weights to be same self.update_network_parameters(tau=1) #this function helps to retrieve actions by adding noise to output network def choose_action(self, observation): self.actor.eval() #get actor in eval mode #convert observation state to tensor for calcualtion observation = T.tensor(observation, dtype=T.float).to(self.actor.device) #get the output from actor network mu = self.actor.forward(observation).to(self.actor.device) #add noise to our output from actor network mu_prime = mu + T.tensor(self.noise(),dtype=T.float).to(self.actor.device) #set back to training mode self.actor.train() #get the final results as array return mu_prime.cpu().detach().numpy() #training our actor and critic network from memory (Replay buffer) def learn(self): #if batch size is not filled then do not train if self.memory.mem_cntr < self.batch_size: return #otherwise take a batch from replay buffer state, action, reward, new_state, done= self.memory.sample_buffer(self.batch_size) #convert all values to tensors reward = T.tensor(reward, dtype=T.float).to(self.critic.device) done = T.tensor(done).to(self.critic.device) new_state = T.tensor(new_state, dtype=T.float).to(self.critic.device) action = T.tensor(action, dtype=T.float).to(self.critic.device) state = T.tensor(state, dtype=T.float).to(self.critic.device) #set netowrks to eval mode self.target_actor.eval() self.target_critic.eval() self.critic.eval() #fetch the output from the target network target_actions = self.target_actor.forward(new_state) #get the critic value from both networks critic_value_ = self.target_critic.forward(new_state, target_actions) critic_value = self.critic.forward(state, action) #now we will calculate total expected reward from this policy target = [] for j in range(self.batch_size): target.append(reward[j] + self.gamma*critic_value_[j]*done[j]) #convert it to tensor on respective device(cpu or gpu) target = T.tensor(target).to(self.critic.device) target = target.view(self.batch_size, 1) #to train critic value set it to train mode back self.critic.train() self.critic.optimizer.zero_grad() #calculate losses from expected value vs critic value critic_loss = F.mse_loss(target, critic_value) #backpropogate the values critic_loss.backward() #update the weights self.critic.optimizer.step() self.critic.eval() self.actor.optimizer.zero_grad() #fetch the output of actor network mu = self.actor.forward(state) self.actor.train() #using formula from DDPG network to calculate actor loss actor_loss = -self.critic.forward(state, mu) #calculating losses actor_loss = T.mean(actor_loss) #back propogation actor_loss.backward() #update the weights self.actor.optimizer.step() #soft update the target network self.update_network_parameters() #since our target is continuously moving we need to soft update target network def update_network_parameters(self, tau=None): #if tau is not given then use default from class if tau is None: tau = self.tau #fetch the parameters actor_params = self.actor.named_parameters() critic_params = self.critic.named_parameters() #fetch target parameters target_actor_params = self.target_actor.named_parameters() target_critic_params = self.target_critic.named_parameters() #create dictionary of params critic_state_dict = dict(critic_params) actor_state_dict = dict(actor_params) target_critic_dict = dict(target_critic_params) target_actor_dict = dict(target_actor_params) #update critic network with tau as learning rate (tau =1 means hard update) for name in critic_state_dict: critic_state_dict[name] = tau*critic_state_dict[name].clone() + (1-tau)*target_critic_dict[name].clone() self.target_critic.load_state_dict(critic_state_dict) #updating actor network with tau as learning rate for name in actor_state_dict: actor_state_dict[name] = tau*actor_state_dict[name].clone() + (1-tau)*target_actor_dict[name].clone() self.target_actor.load_state_dict(actor_state_dict)The most crucial part is the learning function. First, we feed the network with samples until it fills up to the batch size and then start sampling from batches to update our networks. Calculate critic and actor losses and then just soft update all the parameters.
env = OurCustomEnv(sales_function, obs_range, act_range) agent = Agent(alpha= 0.000025, beta =0.00025, tau=0.001, env=env, batch_size=64, n_actions=2) score_history = [] for i in range(10000): obs = env.reset() done = False score = 0 while not done: act = agent.choose_action(obs) new_state, reward, done, info = env.step(act) agent.remember(obs, act, reward, new_state, int(done)) agent.learn() score += reward obs = new_state score_history.append(score)Just after some training, our agent performs very well and exhausts almost complete budget.
Reinforcement Learning Libraries in PythonThere are plenty of libraries offering implemented RL algorithms like –
Stable Baselines
TF Agents
Keras-RL
Keras-RL2
PyQlearning
We will explore a bit on Stable Baselines and how to use them through an example.
Installation
pip install stable-baselines[mpi] import gym from stable_baselines import DQN env = gym.make('MountainCar-v0') agent = DQN('MlpPolicy', env, learning_rate=1e-3) agent.learn(total_timesteps=25000)Now we need an evaluation policy
mean_reward, n_steps = evaluate_policy(agent, agent.get_env(), n_eval_episodes=10) agent.save("DQN_mountain_car_agent") #we can save our agent in the disk agent = DQN.load("DQN_mountain_car_agent") # or load it
Training the Agent
state = env.reset() for t in range(5000): action, _ = agent.predict(state) next_state, reward, done, info = env.step(action) state = next_state env.render()This gives us a rough idea, how to use create agents to train in our environment. Since RL is still a heavily research-oriented field, libraries updates fast. Stable baselines has the largest collection of algorithms implemented with additional features. It is suggestive to start with baselines before moving to other libraries.
Challenges in Reinforcement LearningReinforcement Learning is very easily prone to errors, local maxima/minima, and debugging it is hard as compared to other machine learning paradigms, it is because RL works on feedback loops and small errors propagate in the whole model. But that’s not it, we have the most crucial part which is assigning the reward function. Agent heavily depends upon the reward as it is the only thing by which it gets feedback. One of the classical problems in RL is exploration vs exploitation. Various novel methods are used to suppress this, for example, DDPG is prone to this issue so authors of TD3 and SAC (both are improvements over DDPG) used two additional networks (TD3) and temperature parameter(SAC) to deal with the exploration vs exploitation problem and many more novel approaches are being worked upon. Even from all the challenges, Deep RL has lots of applications in real life.
ConclusionWe learned what is reinforcement learning, how we model problems into RL. Created environments using OpenAI Gym, wrote agents from scratch, and also learned how to use already build RL libraries like stable baselines. Although it has some challenges, it still helps in major fields like Robotics, Healthcare, etc. I hope you gained some knowledge or refreshed some concepts from this guide. Thanks to Phil, Andrej Karpathy, Sudarshan for their marvelous work through books and blogs.
Reach out to me via LinkedIn (Nihal Singh)
The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.
A Guide To Good Personal Hygiene
Believe it or not, the air around you is full of millions of different types of bacteria, viruses, and parasites. These microorganisms use your body as a host to easily develop and reproduce. In this regard, personal hygiene is important. Good personal hygiene can easily reduce the risk of catching an illness from these microorganisms. On the other hand, the lack of it can negatively affect your physical, mental, and emotional health. In this article, you will learn some proven tips to help you develop a good hygiene routine for you and your loved ones’ well-being.
What Exactly is Meant by “Personal Hygiene”? What does it mean to have poor Personal Hygiene?It doesn’t matter whether you do it unconsciously, like picking your nose or ear with your finger, or if you do it habitually, like not cleaning your teeth thoroughly; either way, you’re engaging in poor personal hygiene. Along similar lines, there are certain behaviours that we intentionally engage in just because we feel lazy or don’t perceive their worth in it. For example, not having a shower every day or not feeling that regular hand washing is necessary are examples of such behaviours.
You should practice the following fundamental aspects of good personal hygiene if you wish to reduce the likelihood of contracting an illness and also improve your general health −
Cut your nails short − Hangnails and infected nail beds are only two of the issues that may be avoided by keeping your fingernails and toenails trimmed and in excellent condition. Those who keep their feet clean and dry are less likely to come down with an athlete’s foot.
Take showers regularly − Taking a regular bath is essential for maintaining good hygiene. While showering every day is generally sufficient for most people, there are some exceptions. For example, if you’re into too much physically active work, you may need to take more than one shower daily. A regular bath helps get rid of dirt, sweat, and other debris from the body, and that prevents the spread of germs and bacteria in our bodies. Also, bathing can help you to moisturize your skin and prevent certain skin conditions, such as acne. Plus, you feel great and de-stressed after a long day post taking a bath.
Always be sure to wash your hands − The covid pandemic has taught us one thing: washing hands is the ideal way to protect our bodies from unwanted illnesses.
You should wash your hands −
Before, during, and after preparing food
Before eating food
After going to a toilet
After changing diapers or cleaning your child post toilet
After blowing your nose, coughing, or sneezing
After touching an animal or petting them
After handling garbage
You should also floss − It is best to practice cleaning your teeth with toothpaste after each meal. Brush your teeth at least twice a day and floss at least once a day as a bare minimum. When you brush your teeth, you reduce the number of bacteria that builds up in your mouth, which helps prevent cavities and gum disease. Additionally, flossing aids in the maintenance of robust and healthy gums. The germs that may accumulate and lead to gum disease can go directly to the heart and create extremely significant issues with the valves there. Your teeth may become more mobile if you have unhealthy gums, making it difficult to chew food and consume it appropriately. Visit your dentist at least once every six months for examinations and cleanings to ensure that your smile remains in good health.
Sleep tight − You should aim to get between eight and ten hours of sleep each night so that you may wake up feeling revitalized and prepared to face the day. Your natural defences, including your immune system, might become compromised when you don’t get enough sleep, which can leave you feeling worn down.
To get rid of body odour − you should always put on some deodorant after you get out of the shower if you want to continue smelling clean. Apply deodorant not just to your underarms but also to your clean garments. If you are sweating, putting on perfume is not a good idea since it will just make your body smell worse. First, wash any sweaty clothing you were wearing, then change into fresh clothes, and last put on some deodorant.
Put your attention first on the safety of your meals − It is recommended that anybody who cooks wash their hands both before and after handling raw or cooked food. If you make this practice a part of your routine, you will eliminate the risk of contaminating food in your home. Hands that are not properly cleaned are more likely to attract germs, which is unhealthy. The germs can be transferred to food, which would then make us sick. As a result, food safety must be maintained, and this may be accomplished by maintaining high standards of sanitation and cleanliness.
Wash Your Hair Regularly − Washing your hair needed regularly. You should wash your hair at least three times each week to keep your scalp and hair clean. Our hair is harmed owing to various environmental causes, including pollution, filth, grime, etc. Our hair suffers a lot of damage as a result of all of these things, which causes it to become dry and frizzy. It is essential to wash one’s hair three times a week using a shampoo that does not include sulphate to eliminate this problem. In addition, use a conditioner on your tresses so that they may be healthy and shiny when you’ve finished styling them.
ConclusionIf you know someone who doesn’t seem to care about their appearance or hasn’t showered in a while, this might be a clue that they are sad. People who are feeling sad or melancholy tend to ignore themselves in their daily lives. Some individuals might benefit from hearing about the significance of maintaining adequate personal cleanliness in warding off sickness and being provided with products related to such hygiene. In your conversations, strive to be honest while yet being sensitive. Despite your best efforts, it’s possible that your friend or loved one needs the assistance of a professional. If they don’t make any progress with their hygiene, you should strongly suggest that they see a therapist or a physician.
Update the detailed information about A Guide To Amazon Indexing & How To Check Your Keyword Indexing on the Cancandonuts.com website. We hope the article's content will meet your needs, and we will regularly update the information to provide you with the fastest and most accurate information. Have a great day!