Why did Jimmy stop caring about Chuck?

I'm on s4e4 and I think I missed something(?). Chuck always ment a lot to Jimmy. He got in some kind of apathy right after Chuck's death but as soon as Howard said it might have been his fault Jimmy stopped grieving about Chuck like it even wasn't his own brother. What made me write this post is the scene when Kim cried while Jimmy was reading Chuck's letter in a casual tone. Can someone explain what actually happened to Jimmy?